Part-based multi-task deep network for autonomous indoor drone navigation

Abstract

Two common methods exist for solving indoor autonomous navigation and obstacle-avoidance problems using monocular vision: the traditional simultaneous localization and mapping (SLAM) method, which requires complex hardware, heavy calculations, and is prone to errors in low texture or dynamic environments; and deep-learning algorithms, which use the fully connected layer for classification or regression, resulting in more model parameters and easy over-fitting. Among the latter ones, the most advanced indoor navigation algorithm divides a single image frame into multiple parts for prediction, resulting in doubled reasoning time. To solve these problems, we propose a multi-task deep network based on feature map region division for monocular indoor autonomous navigation. We divide the feature map instead of the original image to avoid repeated information processing. To reduce model parameters, we use convolution instead of the fully connected layer to predict the navigable probability of the left, middle, and right parts. We propose that the linear velocity is determined by combining three prediction probabilities to reduce collision risk. Experimental evaluation shows that the proposed method is nine times smaller than the previous state-of-the-art methods; further, its processing speed and navigation capability increase more than five and 1.6 times, respectively.

Keywords

Aerial robotics monocular vision indoor navigation obstacle avoidance multi-task deep network

Introduction

Recently, off-the-shelf drones have become cheaper, and widely used for tasks such as emergency response (Michael et al., 2012), search and rescue (Bejiga et al., 2017), navigation for transmission-line inspection (Hui et al., 2019) and surveillance (Faessler et al., 2016; Scherer et al., 2012). For safer flight operation, autonomous navigation has attracted increased attention (Kim and Chen, 2015; Achtelik et al., 2009). Most outdoor autonomous flights use GPS (Kahn et al., 2017; Yun et al., 2011); however, this is limited in indoor scenes, posing a challenge for indoor autonomous navigation (Abbott and Powell, 1999).

Obstacle detection and avoidance methods are important steps to achieve safe and practical autonomous drone navigation. The traditional two-step approach first uses vision (RGB-D sensor (Huang et al., 2011), stereo vision (McGuire et al., 2017)) or distance sensors (LIDAR (Ramasamy et al., 2016)) to create a 3D map of the unknown environment, while calculating the relative position of the device in the environment, which is simultaneous localization and mapping (SLAM) (Engel et al., 2014; Weiss et al., 2012). Then, the control commands are calculated so that the drone can infer the flight state of the aircraft from the three-dimensional model. This method is mostly used for ground robots that can carry high-power hardware; however, it is unsuitable for light drones. At the hardware level, these algorithms need improved power, payload, and other hardware, which will increase the cost; and these heavy, expensive sensors limit the applicability of these algorithms on light commercial drones. In addition, SLAM requires intensive calculations to reconstruct the environment 3D model, which is unsuitable for autonomous navigation with high real-time requirements; in real dynamic environments, especially containing untextured surfaces, these algorithms may have unrecoverable errors. To avoid complex hardware design and high-power onboard sensors, we use the drone’s front-facing camera as the input sensor.

Recently, deep learning (DL) has demonstrated advanced performance in a variety of machine vision tasks. Monocular obstacle detection and obstacle avoidance based on deep learning have become a new trend (Choi and Cha, 2019), and impressive results have been obtained (Imanberdiyev et al., 2016; Kim and Chen, 2015; Padhy et al., 2018). However, reinforcement learning has the nature of trial-and-error. The resultant highly complex sample data hinders application in autonomous navigation systems for drones where safety is a key factor. Recently, the learning control strategy has also been studied in simulation (Sadeghi and Levine, 2017), but its applicability in the physical world is limited (Zhu et al., 2017). At present, the supervised learning in the autonomous navigation of drones uses the fully connected layer, which has too many parameters to be trained and is easy to overfit (Lin et al., 2014). This method can be divided into a model learning and a perception task model. Demonstration learning relies on human expert demonstration commands, which weakens applicability (Giusti et al., 2016). At present, the most advanced perceptual task-based unmanned aerial vehicle (UAV) indoor autonomous navigation obstacle avoidance technique divides the single frame image into multiple parts for processing, which leads to slower inference speed. For this reason, in Section 3.2, we propose to divide the feature graph processed by convolution into regions and use convolution to predict the probability; thus, improving processing speed to make the model lightweight and prevent over-fitting. In addition, the linear velocity of the current algorithm is only determined by the prediction of the intermediate image. When the obstacle appears in the left or right image, the linear velocity of the drone should be reduced to mitigate risk. In Section 3.3, we propose to use the three-part prediction of the image to determine the linear velocity.

In this paper, the method of deep learning fusion motion planning policy is used to solve the problem of indoor autonomous navigation of drone based on monocular vision. The main contributions of this work are:

We propose a part-based multi-task deep network, named PMTD-Net, to solve the problem of inference delay in UAV autonomous navigation.

We propose a novel local motion planning policy. This policy modulates the linear velocity and the angular velocity of the drone according to the three navigation probabilities predicted by PMTD-Net to achieve safe autonomous navigation.

We have collected a custom dataset for training PMTD-Net.

We design a model evaluation and actual flight experiments to test the model performance. The results show that PMTD-Net has better performance than previous navigation algorithms.

The rest of this article will cover the following topics. Section 2 provides an overview of research on autonomous navigation and obstacle avoidance in UAVs. Section 3 details the methods of custom data sets, PMTD-Net architecture, and the motion planning policy. Section 4 presents the algorithm comparison experiments and flight experiments in real scenes. Finally, a conclusive review is given.

Related work

Autonomous navigation tasks are highly correlated with obstacle detection and obstacle avoidance. The monocular obstacle detection method in the literature is either based on scene analysis of classical computer vision algorithms or on machine learning.

Recently, the methods introduced by the deep learning-based monocular obstacle avoidance task are mostly based on the perceptual task; that is, the relative state information of the agent and the environment is extracted from the perceptual input. According to different applicable scenarios, we divide autonomous navigation and obstacle avoidance techniques into indoor and outdoor methods. Outdoor autonomous navigation and obstacle avoidance use transfer methods to collect large amounts of data for training models. Giusti et al. (2016) and Smolyanskiy et al. (2017) used the data set collected by hikers carrying cameras to study the problem of autonomous navigation using drones in forest trails and to treat trajectory recognition as a classification task. However, Giusti et al. (2016) only classify the view direction, which is not applicable to highly dynamic environments (Smolyanskiy et al., 2017). To solve this problem, Smolyanskiy et al. (2017) added the classification of lateral migration, but the algorithm was based on the monomial SLAM algorithm for obstacle avoidance, greatly increasing computational complexity. Loquercio et al. (2018) trained the shared convolutional DroNet model based on a dataset collected from cars, and obstacle images collected by a bicycle-mounted camera. The output of the motion planning policy modulates the angular velocity and the linear velocity of the drone according to the model prediction, so that the drone can navigate in an urban environment. However, the experimental results show that the algorithm is less effective than the indoor navigation algorithm (Gandhi et al., 2017) in a narrow indoor environment.

In the most advanced indoor navigation work, Gandhi et al. (2017) and Kouris and Bouganis (2018) divide each image into three parts and input the model for reasoning. Gandhi et al. (2017) performed two-category reasoning on the full image, left half, and right half of each image through the trained CNN model, and merged the prediction results into the deterministic algorithm to obtain the line velocity and the angular velocity. The work of Kouris and Bouganis (2018) is closely related to the work of Gandhi et al. (2017). They perform regression reasoning on the left, middle, and right overlapping windows of each image, and combine the predicted obstacle distance and local navigation strategy to obtain the angular velocity and the linear velocity of the drone. However, these methods need to process the single-frame image multiple times, which greatly reduces real-time performance (Loquercio et al., 2018). To this end, we propose to extract features of a single image by convolution and divide the region of the feature graph obtained by convolution to avoid repeated processing of information. In addition, we replace the fully connected layer with the convolution layer, which can make the model lighter and prevent over-fitting. To the best of our knowledge, we are the first to partition the feature map and replace the fully connected layer with the convolution layer for autonomous navigation. Both Kouris and Bouganis (2018) and Gandhi et al. (2017) attempted to collect large data sets to prevent model overfitting, but none of the data sets were published. In this study, we design lighter and faster algorithms and verify their performance, independent of regression or classification. Because the regression algorithm of Gandhi et al. (2017) requires additional measurement equipment, we customize the classification data set to verify the performance of the algorithm.

Inspired by the pedestrian re-recognition work of Sun et al. (2018) and SPPNet of He et al. (2015), the convolutional layer only extracts features and can share the convolutional layer to speed up inference. We extract the original image features at one time and use convolution to predict the navigable probability of left, middle, and right parts, to accelerate the reasoning speed. Using the custom data set described in Section 3.1 to train PMTD-Net, our local motion planning policy concludes the predicted results of PMTD-Net into a control command that modulates the angular velocity and the linear velocity of the robot, so as to realize autonomous navigation. For specific implementation details see Drone local motion planning for obstacle avoidance section.

Approach

Custom data sets are used to train and test models. To speed up the processing, we propose a PMTD-Net based on feature map region division. The motion planning policy makes drone autonomous navigation more secure.

Datasets

We used the front camera of the Parrot Bebop2 drone to capture more than 25,000 images of different indoor scenes. The front camera can capture full HD 1080p video at 30 fps. To reduce the difficulty of labelling, we control the drone to fly in a straight line (Padhy et al., 2018). Further, we collect multiple trajectories at different angles relative to the obstacle. We manually mark the collected images. Negative samples are those that are near obstacles, meaning that they are unnavigable, marked as 0; positive samples are those that are away from obstacles, meaning that they are navigable, marked as 1, as shown in Figure 1. Since the images come from video sequences, manual annotation is a feasible method.

Figure 1.

Training data collected from different indoor scenes; the corresponding labels are at the bottom.

According to MobileNetv1 (Howard et al., 2017), the amount of model calculation is related to the input image size. Therefore, we set the aspect ratio of the input image to be the same as that of the original image acquired by the front camera of the drone; this helps avoid filling the input image with invalid information. To the best of our knowledge, we are the first to explicitly consider the relationship between the input image and the original image in the autonomous navigation task.

Learning method

The structure diagram of the most currently advanced indoor navigation and obstacle avoidance algorithm based on deep learning (Gandhi et al., 2017; Kouris and Bouganis, 2018) is shown in Figure 2(a). The image observation $X$ in the current task ${X, Y^{1}, Y^{2}, Y^{3}}$ is cropped into three parts ${X^{1}, X^{2}, X^{3}}$ . Then, the same model is trained with the three parts ${X^{i}, Y^{i}}, i = 1, 2, 3$ as independent inputs. The posterior estimation iteration of the training model is described as follows

\begin{matrix} W^{*} = \underset{W}{\arg max} p (W) p (Y | W, X) \\ = \underset{W}{\arg max} Π_{e = 1}^{epochs} Π_{b = 1}^{bm} {p (W_{e, b}) \cdot Π_{n = 1}^{bs} p (Y_{e, b}^{n} | W_{e, b}, X_{e, b}^{n})} . \\ = \underset{W}{\arg max} Π_{e = 1}^{epochs} Π_{b = 1}^{bm} {p (W_{e, b}) \cdot Π_{n = 1}^{bs} (Π_{i = 1}^{3} p (Y_{e, b}^{n, i} | W_{e, b}, X_{e, b}^{n, i}))} \end{matrix}

(1)

Here, $W^{*}$ represents the end of training or the expected optimal weight value, $e, epochs$ respectively represents the times of epoch training and maximum epoch iteration, $b, bm$ respectively represents the times of batch training and the total batch of samples, and $n, bs$ respectively represents the value of sample and batch size in the batch. $p (Y_{e, b}^{n, i} | W_{e, b}, X_{e, b}^{n, i})$ represents the output prediction probability of part $i$ of the $n - th$ sample by the time of training to batch $b$ of epoch $e$ . Since the tasks of the three parts were the same, previous researchers (Gandhi et al., 2017; Kouris and Bouganis, 2018) trained the three parts as independent samples. Equation (1) shows that advanced indoor autonomous navigation and obstacle avoidance tasks (Gandhi et al., 2017; Kouris and Bouganis, 2018) are essentially multiple tasks; because multiple tasks have the same loss function, the author treats them as the same task. However, each frame of the image needs to be processed thrice, which results in tripling the reasoning time.

Figure 2.

Schematic of the algorithm: (a) most advanced indoor navigation algorithm; (b) PMTD-Net.

In order to solve the inference delay problem and reduce the model parameters, we introduce two techniques in PMTD-Net: a part-based strategy, and a strategy using convolution to replace the fully connected layer. PMTD-Net extracts the feature map of an input image based on sharing weights of convolutional layers. We then divide the feature map into three parts according to the part-based strategy, instead of directly dividing the original image into three parts (Ruder, 2017). We propose to use three convolution layers instead of the fully connected layer as classifiers to reduce the model parameters and make the model lightweight. Finally, we use the convolutional layers to classify the feature maps, and output the navigable probability of the three parts, as shown in Figure 2(b).

Because the ResNet model (He et al., 2016a) has good generalization ability and learning ability, we use it to extract features. The training network predicts the voyage probability of the left, middle, and right parts of the image acquired by the front view camera of the drone. The specific structure is shown in Figure 3(a). The original image is an RGB image collected by the Parrot Bebop2 front camera with a resolution of 1920 × 1080. The input image format of our network model PMTD-Net is an RGB image with a resolution of 240 × 135. We resize the original image by one-eighth of the original image size to obtain the input image, as shown in Figure 2(b). The size of the feature tensor $T$ obtained by an 8-layer Resnet structure is 8 × 5 × 128, namely, a width of 8, height of 5, and 128 channels. We divide the feature tensor into feature tensors with a width of 4 for the left, center, and right. We divide the feature tensor $T$ into the feature tensors $G$ with a width of 2 for the left, center, and right, $G = {g^{1}, g^{2}, g^{3}}$ . Using a convolution of size (4, 5, 128) and a step size of (2, 5), the tensors $G$ after division are operated to obtain tensors with a width of 3, height of 1, and depth of 2. Finally, the sigmoid function predicts the traversable probability of the left, middle, and right parts. Compared with the most advanced indoor autonomous navigation algorithm, this structure has two advantages: (1) the final convolution layer replaces the fully connected layer, and realizes weight sharing, reducing the number of parameters and preventing over-fitting; (2) adopting the area division operation not only maintains the area location information, but also reduces the amount of computation of the repeated processing information. The residual block structure of the network (He et al., 2016b) is shown in Figure 3(b), and the dotted line is a skip connection, as shown in Equation (2). The structure can effectively solve the gradient disappearance/explosion problem of the deep network and the network degradation problem when the model is too deep

\begin{matrix} x_{l} = W_{l, 1} f (g (a_{l})) + b_{l, 1} \\ F_{l} = W_{l, 2} f (g (x_{l})) + b_{l, 2} \\ a_{l + 1} = F_{l} + a_{l} \end{matrix}

(2)

Here, $a_{l}, a_{l + 1}$ are the input and output characteristics of the $l$ -level, respectively, and $F_{l}$ is the residual of the $l$ -level, which prevents the attenuation of the gradient during backpropagation and avoids model degradation when the network is too deep. $W, b$ presents the trainable parameters of the model, and $g (•)$ represents the batch normalization function, which is a batch normalization of input data to reduce the gradient disappearance problem in backpropagation, as shown in Ioffe and Szegedy (2015). $Re LU$ function is a linear correction unit, which is $Re LU = max (x, 0) = {\begin{matrix} x, if x \geq 0 \\ 0, if x < 0 \end{matrix}$ .

Figure 3.

(a) PMTD-Net model for accelerating inference speed; (b) unfolded structure of residual block.

Loss function

Each region of the image corresponds to a classification task, and the loss function of the PMTD-Net is the sum of the cross-entropy loss functions of the left, middle, and right regions of an input image, instead of a single classification loss function, as shown in Equation (3)

L_{total} = \sum_{i = 1}^{3} L^{i}

(3)

Here, $L^{1}, L^{2}, L^{3}$ represent the loss functions of the left, middle, and right parts of the input image, respectively. The PMTD-Net model is used to predict three navigable probabilities ${\hat{P}}^{1}, {\hat{P}}^{2}, {\hat{P}}^{3}$ towards the left, middle, and right parts of the input image, respectively; then the three corresponding obstacle probabilities are $1 - {\hat{P}}^{1}, 1 - {\hat{P}}^{2}, 1 - {\hat{P}}^{3}$ . The cross-entropy loss function of each part is

L^{i} = - P^{i} \log ({\hat{P}}^{i}) - (1 - P^{i}) \log (1 - {\hat{P}}^{i}) .

(4)

Here, $i$ presents ${1, 2, 3}$ , which denotes elements in left, middle, and right parts, that is, ${L, C, R}$ ; $L^{i}$ denotes the cross-entropy loss function of part $i$ ; and $P^{i}$ represents the $i$ part sample label, where the positive sample that can be navigable is 1, and the negative sample with obstacles is 0.

We use the Adam optimizer (Kingma and Ba, 2015) as the model’s optimizer, where a starting learning rate of 0.001 and an exponential per-step decay equal to $10 e - 5$ . We trained $300$ epochs, using $128$ small batches, and shuffled each epoch to form different batches. Desktop servers equipped with Intel Xeon e5-2640 processors, 64 GB of memory and four RTX2080ti GPUs, and the Pytorch framework were used for the design and training of the CNN model. In order to clearly visualize the intermediate features extracted by the PMTD-Net model, we show the feature maps of a training image obtained by a CNN, and the final output by PMTD-Net, in Figure 4.

Figure 4.

The feature maps of a training image and the final output by PMTD-Net.

Drone local motion planning for obstacle avoidance

The trained PMTD-Net is an end-to-end model that learns the mapping from the input image to three navigable probabilities toward the left, middle, and right parts of the input image. These three probabilities are fed to a novel local motion planning to conclude into a control command that modulates the drone’s angular velocity $ϕ_{k}$ and the forward linear velocity $ν_{k}$ . The proposed motion planning policy leverages these probabilities to perform continuous adjustments of the drone’s motion.

This motion planning policy settles an important and special matter that has never been considered in the previous algorithms (Gandhi et al., 2017; Kouris and Bouganis, 2018). When a person or other obstacles appear in the left or right part of the forward-looking camera’s field of view, the drone should slow down in time to reduce threats to the obstacles or the drone. To this end, we design the motion planning based on the three-part navigable probabilities, while the previous algorithms are only based on the prediction of the middle part of the input image to modulate the linear velocity. When the above situation occurs, the motion planning policy reduces the linear velocity of the drone in time to avoid threats to the obstacles in the environment.

To eliminate the oscillations that may occur in the motion of the drone, we propose to use a low-pass filter for the linear velocity

v_{k} = (1 - μ) v_{k - 1} + μ p_{k} V_{max} .

(5)

Here, $ν_{k}, ν_{k - 1}$ are the linear velocity of the current moment $k$ of the quadrotor and the $k - 1$ of the previous moment respectively; $μ$ is the weight of the linear velocity filter, which eliminates the oscillation that may occur in the linear velocity of the drone, whose value range is $[0, 1]$ and is set to 0.7 according to experience. $V_{max}$ is the maximum linear velocity of the drone, which can be set to different values according to different scenarios; $p_{k}$ is determined by a combination of the three parts of the navigable probability to adjust the linear velocity, and its value range is $0 \leq p_{k} \leq 1$ , as shown in Algorithm 1.

Algorithm 1. Policy for the parameter $p_{k}$
Input: ${\hat{P}}^{1}, {\hat{P}}^{2}, {\hat{P}}^{3}, P_{th}$ $A [3]$ is an all-zero array
Output: $p_{k}$
1: if ${\hat{P}}^{2} > P_{th}$ then
2: Assign ${\hat{P}}^{1}, {\hat{P}}^{2}, {\hat{P}}^{3}$ greater than $P_{th}$ to the array $A$ in turn
3: $p_{k} = \frac{1}{3} \sum_{m = 0}^{2} A [m]$
4: else
5: $p_{k} = 0$

Here, threshold $P_{th}$ is used to determine whether the predicted sample is a positive or negative sample. The array $A$ , which is cleared before each use, is used to store the predicted probabilities ${\hat{P}}^{1}, {\hat{P}}^{2}, {\hat{P}}^{3}$ greater than the threshold $P_{th}$ .

The low-pass filter for the angular velocity is

ϕ_{k} = (1 - τ) ϕ_{k - 1} + τ Δ ϕ_{k} .

(6)

Here, $ϕ_{k}, ϕ_{k - 1}$ respectively represent the angular velocity of the current moment $k$ of the quadrotor and the angular velocity of the previous moment $k - 1$ ; and $τ$ is the weight of the angular velocity filter, which eliminates the oscillation that may occur in the angular velocity of the drone, whose value range is $0 < τ \leq 1$ . According to experience, it is set to $0.7$ . $Δ ϕ_{k}$ is determined by a combination of the three parts of the voyage probability to adjust the angular velocity, whose value range is $- 1 \leq Δ ϕ_{k} \leq 1$ . When there is no obstacle in the middle part, the drone will go straight; otherwise, the drone will steer in the direction where obstacles are less likely. To the left is the positive direction, to the right is the negative direction. As shown in Algorithm 2:

Algorithm 2 Policy for angular velocity parameter $Δ ϕ_{k}$
Input: ${\hat{P}}^{1}, {\hat{P}}^{2}, {\hat{P}}^{3}, P_{th}$
Output: $Δ ϕ_{k}$
1: if ${\hat{P}}^{2} > P_{th}$ then
2: $Δ ϕ_{k}$ is set to zero
3: else if ${\hat{P}}^{1} < {\hat{P}}^{3}$ then
4: turn right till ${\hat{P}}^{2} > P_{th}$
5: else
6: turn left till ${\hat{P}}^{2} > P_{th}$

In this study, we design a new motion planning policy to reliably control drones from a single front-view camera. An interesting phenomenon is that the PMTD-Net model can generate three navigable probabilities from a single image to modulate the linear velocity and the angular velocity of the drone without the need to design additional algorithms to calculate the distance from the target, such as the vanishing point method (Garcia et al., 2019) and so on. In fact, the success of convolutional networks in monocular depth estimation tasks (Mancini et al., 2017) is well known. This also explains why the PMTD-Net model can modulate the linear velocity based on predicted probabilities.

Results and discussion

In this section, we present the quantitative and qualitative results of the PMTD-Net model. First, we use a set of performance indicators to evaluate the accuracy and speed of reasoning of the PMTD-Net model. We then discuss the drone navigation capabilities and compare them to a set of navigation baselines.

Hardware platform

The Parrot Bebop 2.0 drone is used it in this study for capturing all images and for flight capabilities testing. It is equipped with a 14-megapixel CMOS wide-angle lens (92 degrees) that captures images at 30 frames per second. During the experiment, the drone establishes Wi-Fi communication with the Laptop computer (host) equipped with an Intel Core i5-3320M processor and 4 GB RAM. The main function of the host is to perform PMTD-Net model reasoning and motion planning. The system overview and the main model of the host are shown in Figure 5.

Figure 5.

System overview and the main model of the host.

Prediction results

To evaluate the navigational space prediction, we use the commonly used performance indicators of classification: accuracy Acc and F-1 score (Loquercio et al., 2018). Two other important indicators for evaluating models are the size of the model and the speed of reasoning of the model.

Table 1 compares the performance of the PMTD-Net with the performance of other UAV autonomous navigation algorithms (Gandhi et al., 2017) and the model in which the convolution layers of AlexNet are followed by three fully connected layers of the multi-task model. In addition, we use a constant estimator as a weak baseline, which always has a probability of 1, and a random baseline that gives the predicted value at random. From the experimental results, we can conclude that our algorithm is 80 times smaller than the state-of-the-art architecture, and yet 5 times faster (20 frames per second) while maintaining a fairly good predictive performance. The PMTD-Net model achieves a good balance between processing performance and processing time. In fact, for a drone to respond quickly to unexpected events or dangerous situations, it is necessary to minimize the delay in network reasoning. In addition, because the acquired data is limited, our PMTD-Net model avoids the use of a fully connected layer with a large number of parameters to avoid over-fitting of the network.

Table 1.

Quantitative results of classification, model size, and reasoning speed.

Model	Average Accuracy	F1-score	Parameters	Processing time (fps)
Constant baseline	−	−	−	−
Random baseline	−	−	−	−
AlexNet (Gandhi et al., 2017)	$93.78 %$	$0.9308$	$2.7 e + 7$	$5.6$
AlexNet-3fcs (ours)	$95.43 %$	$0.9379$	$7.3 e + 7$	$10.1$
PMTD-Net (ours)	$96.96 %$	$0.9645$	$2.9 e + 6$	$28.2$

Quantitative results of drone navigation capability

In the actual test, we use the average flight time as performance indicators.

In this section, we use the PMTD-Net model and the drone motion planning policy to perform real-fly evaluations in autonomous drone navigation tasks. We studied four different indoor scenarios, including indoor environments with or without static obstacles, and corridors with or without dynamic obstacles. The configuration of the obstacles in these scenes is different, as shown in Figure 7. We compare our approach to three baselines:

Straight line strategy: Without any feedback, the drone flies in a straight line as a weak baseline.

Human manipulation strategy: The pilot controls the drone based on the video captured by the front view camera of the drone as a strong baseline.

Collision probability minimization strategy: Gandhi et al. (2017) design a classification model based on Alexnet network (Krizhevsky et al., 2012) and provide the prediction results of the model to the decision scheme. The scheme modulates the linear velocity based on the predicted collision probability of the middle part, and it modulates the angular velocity of the drone based on the predicted collision probability in each direction. To estimate the probability of navigation, we used the same data set as this article.

After navigating for five minutes in all test scenarios, we compared the average flight time of all strategies from take-off to collision and minimized human intervention (Table 2). The comparison results show that the proposed method is superior to the autonomous baseline (Gandhi et al., 2017). It can navigate the $4.6 \times$ (average value) time without touching obstacles in the test environment, reaching $0.53 \times$ and $0.63 \times$ flight time of pilots respectively, and exceeding the weak baseline of straight baseline $10.7 \times$ and $7.4 \times$ time in the conference room and corridor scenes, respectively.

Table 2.

Average travelled time before collision of four policies in several environments.

Method	Hallway	Hallway2
	Time (s)	Time (s)
Straight path policy	14.8	11.4
Human pilot	169.4	64.6
Gandhi et al. (2017)	65.1	30.2
PMTD-Net (ours)	121.0	50.8

In order to understand the dynamic process of the drone flight more clearly, we show a set of saved actual flight data. This dataset includes images in flight and control commands transmitted to the drone, as shown in Figure 6. The shape of the corridor where the drone is flying is a left-turn L-shape. Four scenes of the corridor have been attached to the figure. The horizontal axis in the figure represents time. The black solid line in the figure represents the linear velocity, and the gray dotted line represents the angular velocity. It can be seen from the figure that when there is an obstacle in front of the drone, the linear velocity of the drone becomes 0; meanwhile, the drone rotates to the left or right, which has a higher probability of prediction, until the middle part is predicted to be navigable. The signal fluctuates almost vertically, indicating that the system can respond quickly to external changes.

Figure 6.

Dynamic outputs of our system.

Qualitative results on drone navigation

Figure 7 illustrates the input example of the forward-looking camera of the UAV and the steering instructions generated by each method. For our method, when the obstacle is in the middle, the generated command is to stop flying and steer in the direction with greater navigable probability; otherwise, the drone flies in a straight line. As can be seen from the qualitative result diagram, our method is more similar to human control decisions. The prediction results from Gandhi et al. (2017) indicate misjudgments in previously unseen environments.

Figure 7.

Steering instructions generated by each method.

Conclusions

We proposed a deep learning model, named PMTD-Net. The model adopts the part-based strategy to divide the feature map of an input image into three parts. This strategy helps our PMTD-Net speed up model inference. An additional strategy was the replacement of the fully connected layers with convolutional layers as classifiers. Furthermore, we improved the drone motion planning policy; this policy relies on the three navigable probabilities of the input image, not only the prediction probability of the middle part. This strategy greatly improves flight reliability. Through experiments, we proved that our PMTD-Net is nine times computationally smaller than the previous state-of-the-art methods, the processing speed is five times higher, and the navigation ability increased by more than 1.6 times. However, the classification algorithm can easily over-predict the results. Previous reports have proposed an additional regression algorithm to solve this problem. In the future, we will try to solve the problem of fewer texture images in the regression task with deep learning

Footnotes

Appendix

Acknowledgements

We would like to thank Editage () for English language editing.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Scientific Instruments Development Program of NSFC [615278010]; and by Science and Technology Planning Project of Guangdong, China [2017B010116005].

ORCID iD

Xiangzhu Zhang

References

Abbott

Powell

1999 Land-vehicle navigation using GPS. Proceedings of the IEEE 87(1): 145–162.

Achtelik

Bachrach

, et al. (2009) Stereo vision and laser odometry for autonomous helicopters in GPS-denied indoor environments. Proceedings of SPIE 7332: 733219–733228.

Bejiga

Zeggada

Nouffidj

Melgani

(2017) A convolutional neural network approach for assisting avalanche search and rescue operations with UAV imagery. Remote Sensing 9(2): 100–121.

Choi

Cha

(2019) Unmanned aerial vehicles using machine learning for autonomous flight; state-of-the-art. Advanced Robotics 33(6): 265–277.

Engel

Schöps

Cremers

(2014) LSD-SLAM: Large-scale direct monocular SLAM. In: Computer Vision-(ECCV) 2014-13th European Conference (eds Fleet

Pajdla

Schiele

, et al.), Zurich, Switzerland, 6–12 September 2014, Part VII, pp. 834–849. Switzerland: Springer.

Faessler

Fontana

Forster

, et al. (2016) Autonomous, vision-based flight and live dense 3D mapping with a quadrotor micro aerial vehicle. Journal of Field Robotics 33(4): 431–450.

Gandhi

Pinto

Gupta

(2017) Learning to fly by crashing. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017), Vancouver, Canada, 24–28 September 2017, pp. 3948–3955. Piscataway: IEEE.

Garcia

Mittal

Kiewra

Ghose

(2019) A convolutional neural network vision system approach to indoor autonomous quadrotor navigation. In: 2019 International Conference on Unmanned Aircraft Systems (ICUAS), Atlanta, USA, 11–14 June 2019, pp. 1344–1352. Piscataway: IEEE.

Giusti

Guzzi

Cireşan

, et al. (2015) A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters 1(2): 661–667.

10.

Zhang

Ren

Sun

(2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(9): 1904–1916.

11.

Zhang

Ren

Sun

(2016a) Deep residual learning for image recognition. In: 2016 (IEEE) Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, pp. 770–778, USA: IEEE Computer Society.

12.

Zhang

Ren

Sun

(2016b) Identity mappings in deep residual networks. In: Leibe

Matas

Sebe

, et al. (eds) Computer Vision - (ECCV) 2016 - 14th European Conference. Amsterdam: Springer, pp. 630–645.

13.

Howard

Zhu

Chen

, et al. (2017) MobileNets: Efficient convolutional neural networks for mobile vision applications. Available at: https://arxiv.org/abs/1704.04861 (accessed 3 February 2020).

14.

Huang

Bachrach

Henry

, et al. (2017) Visual odometry and mapping for autonomous flight using an RGB-D camera. In: Robotics Research-The 15th International Symposium of Rpbotic and Research (ISRR) (eds Christensen

Khatib

), Arizona, USA, 12–15 September 2015, pp. 235–252. Switzerland: Springer.

15.

Hui

Bian

Zhao

Tan

(2019) A monocular-based navigation approach for unmanned aerial vehicle safe and autonomous transmission-line inspection. International Journal of Advanced Robotic Systems 16(1): 1–16.

16.

Imanberdiyev

Kayacan

, et al. (2016) Autonomous navigation of UAV by using real-time model-based reinforcement learning. In: 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV 2016), Phuket, Thailand, 13–15 November 2016, pp.1–6. Piscataway: IEEE.

17.

Ioffe

Szegedy

(2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015) (eds Bach

Blei

), Lille, France, 6–11 July 2015, pp. 448–456. Brookline: JMLR.

18.

Kahn

Villaflor

Pong

, et al. (2017) Uncertainty-aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182.

19.

Kim

Chen

(2015) Deep neural network for real-time autonomous indoor navigation. arXiv preprint arXiv:1511.04668.

20.

Kingma

(2015) Adam: A method for stochastic optimization. In: Bengio

LeCun

(eds) 3rd International Conference on Learning Representations (ICLR). San Diego, CA, USA.

21.

Kouris

Bouganis

(2018) Learning to fly by myself: A self-supervised cnn-based approach for autonomous navigation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018), Madrid, Spain, 1–5 October 2018, pp.1–9. Piscataway: IEEE.

22.

Krizhevsky

Sutskever

Hinton

(2012) Imagenet classification with deep convolutional neural networks. In: 26th Annual Conference on Neural Information Processing Systems-Volume 1 (NIPS 2012) (eds Bartlett

Pereira

FCN

Burges

CJC

, et al.), Lake Tahoe, Nevada, United States, 3–6 December 2012, pp.1097–1105. New York: Curran Associates Inc.

23.

Lin

Chen

Yan

(2014) Network in network. In: 2nd International Conference on Learning Representations (ICLR 2014) (eds Bengio

LeCun

), Banff, Canada, 14–16 April 2014, pp.1–10. Brookline: JMLR.

24.

Loquercio

Maqueda

Del-Blanco

Scaramuzza

(2018) Dronet: Learning to fly by driving. IEEE Robotics and Automation Letters 3(2): 1088–1095.

25.

Mancini

Costante

Valigi

, et al. (2017) Toward domain independence for learning-based monocular depth estimation. IEEE Robotics and Automation Letters 2(3): 1778–1785.

26.

McGuire

De Croon

De Wagter

, et al. (2017) Efficient optical flow and stereo vision for velocity estimation and obstacle avoidance on an autonomous pocket drone. IEEE Robotics and Automation Letters 2(2): 1070–1076.

27.

Michael

Shen

Mohta

, et al. (2012) Collaborative mapping of an earthquake damaged building via ground and aerial robots. Journal of Field Robotics 29(5): 832–841.

28.

Padhy

Verma

Ahmad

, et al. (2018) Deep neural network for autonomous UAV navigation in indoor corridor environments. Procedia Computer Science 133(2018): 643–650.

29.

Ramasamy

Sabatini

Gardi

Liu

(2016) LIDAR obstacle warning and avoidance system for unmanned aerial vehicle sense-and-avoid. Aerospace Science and Technology 55(August 2016): 344–358.

30.

Ruder

(2017) An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.

31.

Sadeghi

Levine

(2016) Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201.

32.

Scherer

Rehder

Achar

, et al. (2012) River mapping from a flying robot: State estimation, river detection, and obstacle mapping. Autonomous Robots 33(1-2): 189–214.

33.

Smolyanskiy

Kamenev

Smith

Birchfield

(2017) Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017), Vancouver, Canada, 24–28 September 2017, pp. 4241–4247. Piscataway: IEEE.

34.

Sun

Zheng

Yang

, et al. (2018) Beyond part models: person retrieval with refined part pooling (and A strong convolutional baseline). In: Computer Vision-(ECCV) 2018-15th European Conference (eds Ferrari

Hebert

Sminchisescu

, et al.), Munich, Germany, 8–14 September 2018, pp.501–518. Switzerland: Springer.

35.

Weiss

Achtelik

Chli

Siegwart

(2012) Versatile distributed pose estimation and sensor self-calibration for an autonomous MAV. In: IEEE International Conference on Robotics and Automation, St. Paul, Minnesota, IEEE, pp. 31–38.

36.

Yun

Cai

Chen

, et al. (2011) GPS signal enhancement and attitude determination for a mini and low-cost unmanned aerial vehicle. Transactions of the Institute of Measurement and Control 33(6): 665–682.

37.

Zhu

Mottaghi

Kolve

, et al. (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA 2017), Marina Bay Sands, Singapore, 29 May–3 June 2017, pp. 3357–3364. Piscataway: IEEE.