Deep Deterministic Policy Gradient for Navigation of Mobile Robots

Abstract

This article describes the use of the Deep Deterministic Policy Gradient network, a deep reinforcement learning algorithm, for mobile robot navigation. The neural network structure has as inputs laser range findings, angular and linear velocities of the robot, and position and orientation of the mobile robot with respect to a goal position. The outputs of the network will be the angular and linear velocities used as control signals for the robot. The experiments demonstrated that deep reinforcement learning’s techniques that uses continuous actions, are efficient for decision-making in a mobile robot. Nevertheless, the design of the reward functions constitutes an important issue in the performance of deep reinforcement learning algorithms. In order to show the performance of the Deep Reinforcement Learning algorithm, we have applied successfully the proposed architecture in simulated environments and in experiments with a real robot.

Keywords

Deep Deterministic Policy Gradient Deep Reinforcement Learning Navigation for Mobile Robots

1 Introduction

Deep Reinforcement Learning (Deep-RL) is a new technique that is obtaining important applications in many fields such as the control on discrete systems [13 , 25] and more recently in robotics [8 , 23]. Some of the pioneer papers of deep reinforcement learning in robotics were in a manipulation task in a fully observable and stable environment [7], but if the worked environment has obstacles then the degree of difficulty for robot navigation will increase. A solution proposed for Deep-RL techniques is to discretize the actions to reduce the problem difficulty [26, 31]. However, some new publications describe good results achieved using continuous control actions in mobile robot navigation [2, 27].

This work aims to show the performance of Deep-RL in mobile robot navigation with experiments in simulation and in real environments. So, using the Gazebo simulator two environments were designed, Gazebo has many features and functions that helps to analyse the performance of robots with different algorithms and control architectures [6, 11]. After that, we use a real environment to evaluate the robot performance. The mobile robot used on both simulation and application was the Turtlebot3.

One of the objectives that this work will try to achieve is to demonstrate how effective is the Deep Deterministic Policy Gradient algorithm for the purpose of mobile robot navigation, and the analysis will focus in controlling a mobile robot to move from an initial position to a final desired position in a simulated scenario. Another objective will be to execute experiments of the proposed approach in simulated and in real environments, then, for the purpose of generalization two different environments will be employed, one of this environments will have obstacles, so the robot will learn also to avoid collisions with the obstacles; finally, another objective of the paper will be to show how a mapless navigation can work using deep reinforcement learning.

The neural network that was designed for the aforementioned task will have 14 inputs and 2 outputs, as it is depicted in Figure 1. It is expected that the intelligent agent will not collide with any obstacle on its trajectory to the target.

Fig. 1

System definition: The neural network has 14 inputs that come from 10 laser sensor readings, the previous angular and linear velocity, the information of the distance and angle of the mobile robot with respect to the goal. The robot will receive as control signals the angular and linear velocities that are the outputs of the network.

The article is sliced in seven sections. The first one corresponds to an introduction, related works are described in the second section, the theoretical background is introduced in the third section, the tools used in the development of the project are defined in the fourth section, the fifth section explains the methods employed, the sixth section reports the results obtained in the Gazebo simulator and the experiments in the real environments, the seventh section summarizes the conclusions of this work.

2 Related works

Deep-RL has been applied previously to robotic tasks, among these applications, we can mention the works of [2 , 31]. Mnih et al. [16] calculated a value function for future reward using an algorithm called deep Q-network (DQN) for the Atari games, [9 , 30]. It is necessary to outline that the DQN just uses discrete actions when is applied to a problem. In order to extend the DQN to a continuous control, Lillicrap et al. [13] proposed a deep deterministic policy gradients (DDPG) algorithm. This algorithm pave the way to the use of Deep-RL in mobile robot navigation.

Qiang et al. [22] proposed a mapless model by applying a simple but efficient Q-learning based method in mobile robot navigation only taking laser range findings and the target relative position as inputs. This approach not only helps the robot agent to accomplish a navigation task in an environment, but also allows the robot to apply the strategy learned in one environment to another new one.

Dobrevski et al. [4] has adopted an approach for mapless goal-driven navigation optimizing a behavioral policy within the framework of reinforcement learning, specifically the Advantage Actor-Critic (A2C) method. The results are compared with the performance of the standard Turtlebot3 navigation package and it was found that the approach reached a more robust performance.

Mirowski et al. [15] proposed a stacked LSTM architecture for the baseline of the Asynchronous Actor-Critic (A3C) method to navigate in a complex environment with dynamic elements that receive only RGB images.

Chen et al. [1] employed a deep Convolutional Neural Network (ConvNet) to create a direct perception approach, using an input image to map a small number of key perception indicators. It has been observed that this model can work well to drive a car in a very diverse set of virtual environments, though this approach can generalize well to real driving images.

Tai et al. [27] developed a mapless motion planner for a mobile robot and used as input to their system the range findings sensor information and the target position and the output was the continuous steering commands fo the robot. Initially, they employed discrete steering commands on [26]. It was demonstrated that, with the asynchronous Deep-RL method, it is possible to train an agent to arrive to a predetermined target using a mapless motion planner.

Zhu et al. [31] created a model that was fed with the current observation of states and the image of the target as input and has as output an action in a 3D environment, this model reached the objective of applying Deep-RL to drive a mobile robot.

Autonomy for a vehicle to navigate and obtain a good performance in pedestrian-rich environments is a challenging task in robotics. Chen et al. [2] used Deep-RL to propose a technique that can establish what to do and not to do when a robot has to deal with this pedestrian environment respecting social norms and navigating safely.

As in the work of Tai et al. in [27], this paper focuses on developing a mapless motion planner based on low-dimensional range readings. But we use a deterministic approach for the resolution of the navigation problem in mobile robots and the insertion of a dynamic target for the robot in the environments without an asynchronous training. As it is seen, most of the works of Deep-RL only focus their results and application in simulated enviroments. With this, we aim to test the DDPG algorithm approach, our architecture and reward systems for real scenarios and see how effective can be the transfer learning of this knowledge made on simulated environments for the problem of robotic navigation.

3 Deep reinforcement learning

A simplification of the problem of deep reinforcement learning could state that it consists on the control of an agent in an environment trying to maximize its reward function. The deep Q-network (DQN) algorithm formulated in [16, 17] was effective on many Atari video games. This algorithm was able to perform on a human level using only a raw pixel image input to estimate the action of the agent. But despite these accomplishments, the DQN was capable of just using discrete action spaces to solve the complex observation spaces. However, when treating with many tasks in robotics that involves control, it is necessary to make use of continuous action spaces. So another algorithms is necessary to be applied when leading with continuous domain tasks.

3.1 Actor-critic

The policy can be represented by neural networks that updates by gradient either deterministically or stochastically. For both ways, the policy gradients need an estimate of the value function of the current policy. A common approach is to use an actor-critical architecture, shown in Figure 2, which consists of two parts: an actor and a critic [12]. The actor refers to a policy and a critic estimates the value function, which can be Q (s, a) or V (s). In Deep-RL, both actor and critic can be represented by nonlinear neural network function approximators. In summary, it can be said that the actor network is responsible for exploring the agent’s actions in the environment. While the critic network guides the actor network saying whether that action is good or bad. The Actor-Critic algorithm is described in the Appendix 7.

Fig. 2

Actor-Critic architecture.

3.2 Deep deterministic policy gradient

The deep deterministic policy gradient (DDPG) algorithm consists of an actor-critic method that uses approximation functions that can learn continuous action space policies [13]. The algorithm makes use of a neural network for the actor network and other for the critic network. These two networks compute the action prediction for the current state and generate a temporal-difference error signal for each time step. The input of the actor network is the current state, and the output is a real value representing an action chosen for a continuous action space. The output of the critic is simply the estimated Q-value of the current state and the action given by the actor.

The biggest challenge of learning in continuous action space is exploration. In order to overcome this challenge is necessary to construct an exploration policy μ′ by adding a noise sampled from a noise process $N$ to the actor network policy, defined as: $μ^{'} = μ (s_{t}) + N$ (1) where $N$ can be chosen in a way to suit the environment. Being the Ornstein-Uhlenbeck process [29] the most used to generate temporally correlated exploration efficiency in physical control problems.

Training and evaluating a policy function coming out of the actor network and a value function coming out of the critic network, with thousands of simulated trajectories temporally correlated could lead to the introduction of huge amounts of variance in the approximation of a true Q-function. It was suggested as a solution of this issue to use a replay memory to store the experiences of the agent during the training [24]. This means, saving the states, actions, rewards and new states that the agent explored during the episode. And then, randomly sampling experiences to use for learning in order to break the temporal correlations within different training episodes.

The experience replay allows the intelligent agent to learn from recent memories, increasing the learning speed and breaking undesirable temporal correlations. Even with a short memory it is possible to see a substantial improvement in the agent performance. Despite the application of a replay memory delays the agent learning, a better performance is achieved.

Besides the replay memory used, it was necessary a target network to generate targets for the temporal-difference error that can regulate the learning and improve stability. The target network contains a copy of the actor and critic, however, with “soft" updates. This means that the target network do not copy directly the weights of the actor and critic network. The weights, represented by θ, of the target network are then updated by: $θ^{'} \leftarrow τ θ + (1 - τ) θ^{'}$ (2) with τ ⪡ 1. Where the target is represented by the apostrophe. So the target values of the network change slowly and allows an improvement in the stability of the learning.

4 Experimental setup

In this section is describe the robots, tools and environments used in the project.

4.1 PyTorch

Deep learning frameworks usually have problems achieving good usability and speed simultaneously. PyTorch is a machine learning library that shows that achieving these two goals is indeed possible [19]. That framework provides an imperative programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. Therefore, PyTorch has become a popular tool in the deep learning research community by combining a focus on usability with careful performance considerations.

4.2 ROS

The robot operation system (ROS) is a meta-operating system that contributes to share, recycle and write software for robots. ROS [21] has different libraries and functions that are ready to use and simplifies the process of making software for robotic applications. ROS manages different processes that are related to nodes, the nodes have communication among them with topics, so it is possible to read or send information publishing or subscribing to the nodes of the process.

4.3 Gazebo

Although robotics experimentation is essential in robotics [3 , 28], simulation a lot of times is the firs step for the study and analysis of a system. There are multiple simulators, but ROS integrates in its architecture the Gazebo [6] simulator that makes possible to simulate environments easily and with the advantage of having an active community. One of the main advantages of Gazebo is that the parameters and structure in the simulated robot are identical to the real robot and the integration with the ROS framework makes things easier in the process of migrating from the simulation to the application in a real environment.

4.4 Turtlebot

Turtlebot is an educational mobile robot that is fabricated by the company Robotis, for the experiments of this article it is used the Burguer model of the third version of these series of robots, as it is depicted in Figure 3.

Fig. 3

Real Turtlebot3 version Burger.

The robot has 2 DYNAMIXEL motors, a laser LiDAR sensor with 360 degrees, an IMU sensor that measures the orientation of the robot. The low level control is made with a OpenCR1.0 board and a Raspberry Pi 3 microprocessor. The Table 1 shows all hardware specification of the Turtlebot3 version Burger.

Table 1

Hardware specification of the turtlebot3 burger

Items	Specification
Maximum translational	0.22 m/s
velocity
Maximum rotational velocity	2.84 rad/s (162.72 deg/s)
Maximum payload	15kg
Size (L x W x H)	138mm x 178mm x 192mm
Weight	1kg
SBC (Single Board Computers)	Raspberry Pi 3 Model B
	and B+
Actuator	DYNAMIXEL XL430-W250
LDS(Laser Distance Sensor)	360 Laser Distance Sensor
	LDS-01
IMU	Gyroscope 3 Axis
	Accelerometer 3 Axis
	Magnetometer 3 Axis

4.5 Simulation environments

Two environments for the simulations were constructed in Gazebo. The first one can be observed in Figure 4a, an area without obstacles and free just limited by walls was designed for this first environment. A negative reward is generated and the episode finishes when the robot has a collision with an obstacle or a wall. In the second environment 4 obstacles were allocated as it is depicted in Figure 4b, the addition of these fixed obstacles will produce a more challenging scenario for the robot.

Fig. 4

Environments created for the experiments in the Gazebo simulator.

The target in the simulated environments is placed in a random position on the map. With this, the scenario is transformed to be more dynamic and prevent the robot to memorize a strategy for the navigation on the map.

4.6 Real environments

After training the DDPG network through simulation, it will be tested in a real scenario. The Turtlebot3, version Burger, will be used to perform this test. Some of the necessary inputs of the network in the real environment were obtained by image processing. The first real environment is shown in Figure 5a and resembles the first simulation with no obstacles. The second environment is shown in Figure 5b and it presents a complex environment similar to the second used on simulation. The targets of both images were represented by the yellow points.

Fig. 5

Environments created for the DDPG test on the real world.

5 Methodology

The intention of this work is to propose a system for a mobile robot in order to plan its movements without any map knowledge on the environment. The equation of its translation is: $v_{t} = f (x_{t}, p_{t}, v_{t - 1})$ (3) The variable x_t is the observation derived from the raw sensor information, p_t is the relative position of the target, and v_t-1 is the velocity of the mobile robot in the previous step. These aforementioned variables represent the current state s_t of the mobile robot. This model helps to provide the control signals that the robot will have, given its current state. Also, the robot needs to have access to the information of the input data with a good frequency to control its movements during the navigation to avoid collisions and reach its target. This method was previously studied by Jesus et al. in [10] and [27].

5.1 Network structure

Once states and actions of the system have been defined, a DDPG network was designed to tackle the problem. The two networks that are part of the architecture of DDPG have a set of 14 inputs in which are included the laser range sensor readings, previous angular and linear velocities and the position and orientation of the target as it is shown in Figure 6. The laser range readings are between -90 and 90 degrees in relation to the robot. The outputs of the network are angular and linear velocity that will be be the control actions of the robot.

Fig. 6

Inputs and outputs from the networks.

The network structure of the DDPG is shown in Fig 7. The actor-network has as input the current state of the mobile robot followed by 3 fully-connected neural networks layers with 512 nodes. This input of the networks is transformed on the linear and angular velocity that will be the commands sent to the motor of the mobile robot. The angular velocity range is constrained between (-1, 1) and the hyperbolic tangent function (tanh) is used as activation function. For the linear velocity range, that is constrained between (0, 1), the sigmoid function was used. As there is no laser readings in the back of the robot, the backward move is not necessary. The output actions are then multiplied with two hyperparameters to decide the final linear and angular velocity executed by the mobile robot. It was used as maximum linear velocity 0.22 m/s and maximum angular velocity 2 rad/s on the Turtlebot3 robot version Burger. In the critic-network, the Q-value of the state and the actual action are predicted. Using only 3 fully-connected neural networks layers to process the input. Where the action, output of the actor-network, is concatenated in the second layer of the neural network. The Q-value and the value of the current state is activated through a linear activation function: $y = kx + b$ (4) where x is the input of the last layer, y is the predicted Q-value or the predicted value of the current state, and k and b are the trained weights and bias of this layer, respectively.

Fig. 7

DDPG network structure model.

The DDPG network that was implemented for the navigation of mobile robots followed the algorithm described in [13] for the network and it is possible to see the implementation in GitHub (https://github.com/dranaju/project). For the noise it was used the Ornstein-Uhlenbeck process [29]. The hyperparameters for the architecture and optimization are presented in the Table 2.

Table 2

Hyperparameters used on the DDPG network on training a agent for navigation of the mobile robot Turtlebot3 Burger

Hyperparameter	Value
Observation state dimension	14
Continuous action dimension	2
Action maximum linear velocity	0.22 m/s
Action maximum angular velocity	2 rad/s
Replay memory buffer size	200000
Hidden layers	3
Optimizer	Adam
Exploration	Ornstein-Uhlenbeck
Noise (σ)	(0.99, 0.2)
Noise (θ)	0.15
Learning rate (α)	1e - 3
Batch size	256
Discount factor (γ)	0.99
Soft-update (τ)	1e - 3
Steps by episode	500
Training episodes env-1	200
Training episodes env-2	2000
Hardware	CPU

5.2 Reward function

With the simulated environments previously defined, it is now possible to simulate a mobile vehicle in the task of navigation. Firstly, it is needed to specify the reward and penalty system to the Deep-RL network. The rewards and penalties given to the agent are just numbers of a function model of how it wanted the agent to behave, it can be based on empirical knowledge and created during the process of solution of the problem. Thus, the actor network will receive this rewards that will be past to the critic network and it will perform a feedforward and backpropagation step in order to learn the hyperparameters to a Q-value.

There are four different conditions for the reward system that presented better results for the resolution of the problem and are the following: $r (s_{t}, a_{t}) = {\begin{matrix} r_{arrive} if d_{t} < c_{d} \\ r_{collide} if \min_{x} < c_{o} \\ c_{r 1} (d_{t - 1} - d_{t}) if (d_{t - 1} - d_{t}) > 0 \\ c_{r 2} if (d_{t - 1} - d_{t}) \leq 0 \end{matrix}$ (5)

If the robot gets to the target through threshold checking c_d, a positive reward (r_arrive) is given, but if the robot collides with an obstacle through a minimum range readings checking, a negative reward (r_collide) is given. Both conditions are sufficient to end the training episode. Otherwise, the reward is based on the distance difference from the target compared to the last time step (d_t-1 - d_t). If this difference is positive, the reward given is the distance traveled multiplied by the hyperparameter (c_r1), and if the distance is negative, the hyperparameter (c_r2) is used. This motivates the mobile robot to get closer to the target position, and encourages to avoid the obstacles in the environment.

6 Results

This section will describe and discuss the obtained results. The collected data are related to the obtained reward from the artificial neural network of the project. With these data, it is possible to see what is the learning degree of the agent in the environment, because the obtained reward is intrinsically attached to the agent’s performance on the environment that it must go. All the training environments were arranged by ROBOTIS, however, some changes were done on the source code of the Gazebo simulation in order to use the simulated mobile robot and the reward function defined on this paper.

6.1 Simulation results

A mobile robot with a DDPG network was trained in order to make the experiments in the environments shown in Figure 4, where the goal of the mobile robot is to get to the target. The initial test used the first environment defined. It is shown in Figure 8 a sequence of frames of the robot to the target. We can observe how the robot starts in an initial position, distant from the target, navigating in order to reach the target.

Fig. 8

Images sequence in the first simulated environment of the experiment.

The training results of the reward function of the first environment are shown in Figure 9. On the first environment takes less them 10 episodes for the network to converge to an optimal reward. This rapid convergence is done because the environment is simple for the algorithm to lead.

Fig. 9

Rewards of the first simulated environment.

In Figure 9 the x axis represents the past episodes on the simulation, an episode is defined when the mobile robot arrives to the target in the map or collides with some obstacle. The y axis in Figure 9 represents the total value of the reward that the robot received on the episode. The reward, with the blue color, has a wide range of variance. It was decided to use a moving average filter to improve the visualization of the obtained results.

After the mobile robot has been trained on the first environment, the experiment was done in the second environment. It is shown in Figure 10 a sequence of the actions made by the Turtlebot from an initial position until it could arrive to the target after the training episodes.

Fig. 10

Images sequence in the second simulated environment of the experiment.

The reward function results of the training process for the second environment are shown Figure 11. Comparing this results with the last environment, we can observe that it needed more episodes so the robot could present good results. It is noticed that with a more complex environment, exists the possibility that an agent could take a longer time to get a good performance. And for the real reward in the second environment, it is possible to see more noise in the results. This noise can be given because of the introduction of obstacles in the environment. But with the filtered reward, it shows that the reward converges to a optimal value.

Fig. 11

Rewards of the second environment.

6.2 Experiments with the turtlebot robot

After the training and validation of each simulation environment, there were used the two real environments, shown in Figure 5, for the testing of the DDPG network in the real world. Once, the target defined on the first real environment, it was executed the DDPG network on the Turtlebot3 robot with no complications, and the code and parameters utilized in the Gazebo simulation were the same in the experiments with the real robot. Furthermore, it was necessary to use a camera on the ceiling to obtain the target and robot relative position information and deliver it to the DDPG network. Figure 12 shows by frames the trajectory made by the robot to complete the task in the first environment. The trajectory tracks were made with the camera. In the graph, shown in Figure 13, it is possible to see the trajectory made by the network to arrive to the target. The robot starts at the point (0, 0) and makes a trajectory to get to the target. It is possible to see that the network was capable to get to the defined target in the real environment.

Fig. 12

Images sequence of the experiment on the Turtlebot3 in the first real environment.

For the final test of the network, it was used the second real environment.In Figure 13 is shown by frames the trajectory made by the robot to complete the task in the second environment. In the graph, shown in Figure 15, the robot starts at the point (0, 0) and even with obstacles the network was capable to reach the target.

Fig. 13

Trajectory of the Turtlebot3 in the first real environment.

Fig. 14

Images sequence of the experiment on the Turtlebot3 in the second real environment.

Fig. 15

Trajectory of the Turtlebot3 in the second real environment.

The Turtlebot3 robot was able to accomplish both tasks given in the real-world environments. This shows how effective can be the DDPG network, trained in a simulation environment, in completing complex tasks on physical environment.

7 Conclusion

This paper explains how to apply a neural network (DDPG) to tackle the problem of mobile robot navigation through continuous control in a virtual environment. Thus, a deep reinforcement learning network algorithm was capable of executing with a good performance a navigation task. The defined goal for the robot was that it could arrive to a final desired position in two different simulated environments. All the network structure created was first applied on the Gazebo simulation environments with success.

Once, the training results finished satisfactorily in the simulation environments, the experiments were done in the two real environments replicating the good performance that was achieved in simulation making possible for the robot navigate safely to a final location while avoiding obstacles in its trajectory.

One of the conclusions that points out this article is that DDPG networks can be used in continuous control for robotics. An important aspect that can influence the performance of the algorithm is to design a good reward function for the problem that we have to cope. Another conclusion of the article is that the proposed architecture provides to the mobile robot a mean to navigate in complex scenarios both in simulation and in real ones excluding the possibility of having previous previous knowledge about the navigation environments that has as a consequence a mapless navigation. The deep deterministic policy gradients algorithm paves the way for an interesting solution for the use of machine learning in robotics control.

We could state that a limitations of the approach is the training of the algorithm usually takes a lot of time, the literature of Deep RL is working actively to solve this problem, and we hope that this issue could be circumvented. Finally, in a future work, we are planning to apply and analyze the performance of DDPG network in more complex tasks and environments, such as real dynamic environments, that have pedestrians for example. Another proposal is to experiment with algorithms that will have to control a system with multiple agents.

Footnotes

Acknowledgment

We thank all participants of VersusAI by sharing ideas and thoughts in the area of Robotics and Artificial Intelligence: Alisson H. Kolling, Eduardo Santana and Victor A. Kich.

Appendix

References

Chen

, Seff

, Kornhauser

and Xiao

, Deepdriving: Learning affordance for direct perception in autonomous driving, In Proceedings of the IEEE International Conference on Computer Vision (2015), 2722–2730.

Chen

Y.F.

, Everett

, Liu

and How

J.P.

, Socially aware motion planning with deep reinforcement learning, In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 1343–1350. IEEE, (2017).

da Silva

R.M.

, Cuadros

M.A.d.S.L.

and Gamarra

D.F.T.

, Comparison of a backstepping and a fuzzy controller for tracking a trajectory with a mobile robot. In International Conference on Intelligent SystemsDesign and Applications, pages 212–221. Springer, (2018).

Dobrevski

and Skocaj

, Map-less goal-driven navigation based on reinforcement learning, In 23rd Computer Vision Winter Workshop, (2018).

Dos Reis

D.H.

, Welfer

, De Souza Leite Cuadros

M.A.

and Gamarra

D.F.T.

, Mobile robot navigation using an objectrecognition software with rgbd images and the yolo algorithm, Applied Artificial Intelligence 33(14) (2019), 1290–1305.

Fairchild

and Harman

T.L.

, ROS Robotics By Example, Packt Publishing Ltd (2016).

, Lillicrap

, Sutskever

and Levine

, Continuous deep q-learning with model-based acceleration, In International Conference on Machine Learning (2016), 2829–2838.

, Holly

, Lillicrap

and Levine

, Deep reinforcement learning for robotic manipulation with asynchronous offpolicy updates, In 2017 IEEE International Conference on Robotics and Automation (ICRA), 3389–3396. IEEE, (2017).

Hausknecht

and Stone

, Deep recurrent q-learning for partially observable mdps, In 2015 AAAI Fall Symposium Series, (2015).

10.

Jesus

J.C.

, Bottega

J.A.

, Cuadros

M.A.

and Gamarra

D.F.

, Deep deterministic policy gradient for navigation of mobile robots in simulated environments. In 2019 19th International Conference on Advanced Robotics (ICAR), 362–367. IEEE, (2019).

11.

Joseph

, Mastering ROS for robotics programming, Packt Publishing Ltd (2015).

12.

Konda

V.R.

and Tsitsiklis

J.N.

, Actor-critic algorithms, In Advances in neural information processing systems, (2000), 1008–1014.

13.

Lillicrap

T.P.

, Hunt

J.J.

, Pritzel

, Heess

, Erez

, Tassa

, Silver

and Wierstra

, Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971, (2015).

14.

Mahmood

A.R.

, Korenkevych

, Vasan

, Ma

and Bergstra

, Benchmarking reinforcement learning algorithms on real-world robots, arXiv preprint arXiv:1809.07731, (2018).

15.

Mirowski

, Pascanu

, Viola

, Soyer

, Ballard

A.J.

, Banino

, Denil

, Goroshin

, Sifre

, Kavukcuoglu

, et al., Learning to navigate in complex environments, arXiv preprint arXiv:1611.03673, (2016).

16.

Mnih

, Kavukcuoglu

, Silver

, Graves

, Antonoglou

, Wierstra

and Riedmiller

, Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602, (2013).

17.

Mnih

, Kavukcuoglu

, Silver

, Rusu

A.A.

, Veness

, Bellemare

M.G.

, Graves

, Riedmiller

, Fidjeland

A.K.

, Ostrovski

, et al., Human-level control through deep reinforcement learning, Nature 518(7540) (2015), 529–533.

18.

Nachum

, Norouzi

, Xu

and Schuurmans

, Trustpcl: An off-policy trust region method for continuous control, arXiv preprint arXiv:1707.01891, (2017).

19.

Paszke

, Gross

, Massa

, Lerer

, Bradbury

, Chanan

, Killeen

, Lin

, Gimelshein

, Antiga

, et al., Pytorch: An imperative style, high-performance deep learning library, In Advances in Neural Information Processing Systems, (2019), 8024–8035.

20.

Pfitscher

, Welfer

, Do Nascimento

E.J.

, Cuadros

M.A.d.S.L.

and Gamarra

D.F.T.

, Article users activitygesture recognition on kinect sensor using convolutional neural networks and fastdtw for controlling movements ofa mobile robot, Inteligencia Artificial 22(63) (2019), 121–134.

21.

Pyo

, Cho

, Jung

and Lim

, Ros robot programming, Seoul, ROBOTIS Co, (2015).

22.

Qiang

, Nanxun

, Huican

and Heng

, Amodel-free mapless navigation method for mobile robot using reinforcement learning, In 2018 Chinese Control And Decision Conference (CCDC), pages 3410–3415. IEEE, (2018).

23.

Richter

, Orosco

R.K.

and Yip

M.C.

, Open-sourced reinforcement learning environments for surgical robotics, arXiv preprint arXiv:1903.02090, (2019).

24.

Schaul

, Quan

, Antonoglou

and Silver

, Prioritized experience replay, arXiv preprint arXiv:1511.05952, (2015).

25.

Schulman

, Moritz

, Levine

, Jordan

and Abbeel

, High-dimensional continuous control using generalized advantage estimation, arXiv preprint arXiv:1506.02438, (2015).

26.

Tai

and Liu

, Towards cognitive exploration through deep reinforcement learning for mobile robots, arXiv preprint arXiv:1610.01733, (2016).

27.

Tai

, Paolo

and Liu

, Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation, In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 31–36. IEEE, (2017).

28.

Tello Gamarra

D.F.

, Piccinini Legg

, de Souza Leite Cuadros

M.A.

and Santos da Silva

, Sensory integration of a mobile robot using the embedded system odroid-xu4 and ros. In 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE), (2019), 198–203.

29.

Uhlenbeck

G.E.

and Ornstein

L.S.

, On the theory of the brownian motion, Physical Review 36(5) (1930), 823.

30.

Van Hasselt

, Guez

and Silver

, Deep reinforcement learning with double q-learning, In Thirtieth AAAI Conference on Artificial Intelligence, (2016).

31.

Zhu

, Mottaghi

, Kolve

, Lim

J.J.

, Gupta

, Fei-Fei

and Farhadi

, Target-driven visual navigation in indoor scenes using deep reinforcement learning, In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 3357–3364. IEEE, (2017).