RegRL-KG: Learning an L1 regularized reinforcement agent for keyphrase generation

Abstract

Keyphrase generation (KG) aims at condensing the content from the source text to the target concise phrases. Though many KG algorithms have been proposed, most of them are tailored into deep learning settings with various specially designed strategies and may fail in solving the bias exposure problem. Reinforcement Learning (RL), a class of control optimization techniques, are well suited to compensate for some of the limitations of deep learning methods. Nevertheless, RL methods typically suffer from four core difficulties in keyphrase generation: environment interaction and effective exploration, complex action control, reward design, and task-specific obstacle. To tackle this difficult but significant task, we present RegRL-KG, including actor-critic based-reinforcement learning control and L1 policy regularization under the first principle of minimizing the maximum likelihood estimation (MLE) criterion by a sequence-to-sequence (Seq2Seq) deep learnining model, for efficient keyphrase generation. The agent utilizes an actor-critic network to control the generated probability distribution and employs L1 policy regularization to solve the bias exposure problem. Extensive experiments show that our method brings improvement in terms of the evaluation metrics on five scientific article benchmark datasets.

Keywords

Keyphrase generation natural language processing (NLP)natural language generation (NLG)reinforcement learning exposure bias

1. Introduction

Keyphrase generation is a natural language generation task. It partly preserves source text as present keyphrases or chooses the absent content from a pre-defined vocabulary. By distilling the key information of a source text into a set of succinct keyphrases, keyphrase generation serves as the basis for many downstream tasks, including document clustering [12, 8], information retrieval [27, 2], and text summarization [28].

Recently, the field of keyphrase generation study has been further advanced by incorporating the maximum likelihood estimation (MLE) criterion on the final generated distributional probability to train an encoder-decoder deep learning network [18]. Although widely used, MLE-based methods tend to generate universal content [34]. Most of the MLE-based techniques either focus on one limited aspect or rely on an additional loss to enforce some constraints on the process of keyphrase generation. MLE-based models are often prone to suffer from the exposure bias problem [19, 13], i.e., the model is trained conditioned on the ground-truth keyphrases while predicts the next keyword conditioned on its previous generated words. In addition, the MLE criterion is operated on the keyword-level, without considering the instant feedback or long-term effect on subsequent keywords in one keyphrase. Considering a formerly predicted word has a significant influence on the following predictions, the quality of the target keyphrase will be improved by selecting an optimal word at each key step. On the contrary, a bad estimate in one step may lead to divergence of the whole generation process. The error brought by every keyword accumulates and deteriorates the consecutive keyphrases.

Unlike MLE working at the keyword level, reinforcement learning can enable the policy to react instant feedback rapidly and maximize the expected long-term total reward. Another primary driving force behind the explosion of RL in keyphrase generation is its integration with powerful nonlinear function approximators like deep neural networks. This partnership with deep learning methods, often referred to as Deep Reinforcement Learning (DRL), has enabled RL to successfully extend to tasks with high-dimensional input and action spaces. However, designing a reliable keyphrase generation agent is not trivial and often challenging. With the growing need for the keyphrase generation method to handle more complex aspects, it will be much challenging and laborious to build a fully rule-based keyphrase agent, which requires heavy domain expertise. The main disadvantages of rule-based reward models are two-fold: 1) It is difficult to define a comprehensive list of rules. There is a long list of keyphrase generation rules or evaluation criteria in literature, such as F1@5, F1@M, duplication ratio and the average keyphrase number [32, 31]. Each of them focuses on one single aspect. 2) It is difficult to balance different rules. The objective evaluation of keyphrase generation is multi-criteria, where good ratings on one criterion do not necessarily imply good ratings on another. Instead of laboriously designing rule-based reward functions, researchers directly learn the reward model in a data driven manner. It avoids unnecessary or incorrect constraints enforced by human specialists. Data driven RL-based keyphrase generation methods have been proposed in recent studies [4, 24], but the models are often unstable and generate large training variance. Moreover, widespread adoption of data driven reinforcement techniques to the natural language generation task is still limited by four major challenges: environment interaction and effective exploration, complex action control, reward design, and task-specific obstacle. First, RL requires direct interaction with the environment and effective exploration to find good policies as well as avoid converging prematurely to local optima. Nevertheless, environment interactions and effective exploration remain key challenges for DRL in complex environments with large continuous state and action spaces. Then, the complexity of action control comes from the holistic probability distribution generation. Note that the magnitude of actions can reach exponential increments on the average length of a source text and vocabulary size. Such high-dimensional action spaces create challenges to target selection and policy control. Third, RL algorithm needs a predefined reward signal to guide the optimization process. Bootstrapping errors of an inappropriate reward function will deviate the divergence of the objective metrics or feedback less information, otherwise yielding poor performance. Designing a keyphrase reward is not trivial, as it is equivalent to building an evaluative criterion that can judge how well the agent performs its task. A fitness metric can consolidate returns across an entire episode and make RL robust to long time horizons. Finally, despite competing deep learning approaches appearing for KG, RL adaption with the deep learning model is still a severe hindrance in the KG scenario, especially when confronted with the diverse properties of a keyphrase. Meanwhile, we also notice that, for a deep learning-based Seq2Seq model, the training of generation at the next keyword depends on the teacher forcing mechanism [30] rather than on the past predicted keywords. However, during testing, the model has to make predictions based on its previous predictions. The exposure bias inherent in deep learning-based methods tends to deteriorate the subsequent generation quality over time, leading to significant drifts and divergence of the objective metrics generation goes on [13].

To tackle these issues, some reinforcement algorithms are explored to coordinate the supervised Seq2Seq model [4, 24]. Different from the reinforcement algorithm which calculates the metric scores of the entire target keyphrases as rewards, here we first design an actor-critic (AC) framework with a generalized advantage estimator (GAE) [22] to provide a reward to each keyword in the predicted keyphrase and add an L1 policy regularization during the RL training, without explicit domain knowledge. The advantage is that the model can be trained on large volumes of available source text without requiring expensive domain expertise, making it more generalizable. Together with the advantage function from the actor-critic network and the L1 policy regularization, our algorithm updates the Seq2Seq model with a smooth gradient estimate and substantially reduces the training variance. Specifically, a backbone deep learning Seq2Seq network is utilized to extract basic features using MLE criteria to model keyword accuracy. Then we operate AC on the keyphrase generation policy. The actor network selects its action conditioned on the varying action space, while the critic is trained with the guidance of the target keyphrase. Reinforcement learning (RL) methods can operate on the final generation policy, which effectively encodes specific attitude along the generation axis. What is more, due to the natural property of the separated exploration and generation phases in RL, a policy regularization loss is introduced to reduce the compact from the exposure bias problem. Our intuition behind the network design is that high-accuracy and low variance predictions should be preferred. For higher-confidence predictions, the RegRL-KG introduces the relative reward signals by trial and error, which captures the global temporal return and the dependency between the iterative generation and exploration stages. The global temporal return can evaluate the boosted keyphrase generation quality after each backward loss. The dependency is used to judge if the learned policy strategy in the training phrase can choose a more robust and powerful action than the exploration phase. RL provides a more stable and robust representation with the designed relative reward.

The distinctive points of our approach are summarized as follows:

We present an L1 regularized actor-critic based keyphrase generation method, namely RegRL-KG, which can operate direct manipulation on the generated distribution probability in a deterministic environment.

L1 policy regularization is introduced to alleviate the exposure bias problem and reduce the variances between the exploration and generation phases during training.

We demonstrate that RegRL-KG can boost the task metrics and achieve state-of-the-art performance on five standard science article datasets.

In the rest of the paper, we discuss related keyphrase generation and reinforcement learning methods in Section 2. Section 3 gives a brief formalization of the reinforcement background. Section 4 presents the proposed keyphrase generation method RegRL-KG. Section 5 provides experiment implementation. Section 6 provides experiment results and discusses the impact of the presented contributions. Section 7 concludes the paper and indicates the potential future work.

2. Related work

Keyphrase generation task can be categorized into keyphrase extraction and generation types. One solution is keyphrase extraction [33, 17, 10]. Other areas of keyphrase generation incorporate additional generation techniques like copy mechanism, coverage mechanism, orthogonal regularization, semantic coverage, and exclusion mechanisms to further improve the keyphrase generability [32, 7, 6, 31]. Another exciting thread of research is the extension of reinforcement learning into deep learning settings [4, 24].

2.1 Keyphrase extraction and generation

Early deep extraction keyphrase approaches began from selecting relevant keywords from the source text [33, 17, 10]. The drawback of the extraction keyphrase approaches comes from that it can not generate absent information that does not appear in the source text.

The keyphrase generation methods are discovered using standard backpropagation training via MLE criterion [18, 26, 3, 6], which generates target keyphrases with the content of preserved present content from the source text and the absent information from a pre-defined vocabulary. Meng et al. [18] applied an attentional encoder-decoder architecture and accomplish to learn the generated keyword distribution of present and absent representations by weight sharing and copy mechanisms. To improve model performance, various control strategies like coverage mechanism [26], orthogonal regularization [3], target encoding strategies and semantic coverage [3], and exclusion mechanisms [6] have been explored to bridge the gap across the predicted probabilities and the ground-truth. Some alternative methods directly add an auxiliary component, e.g., variational autoencoder [29] or generative adversarial network [24] to assist in keyphrase generation. Similar scalability and competitive results were demonstrated by Chen et al. using title information to guide keyphrase generation [7]. The up-to-date keyphrase generation method is One2Set [31]. The authors suggest control codes and a K-step target assignment mechanism via bipartite matching to generate keyphrases. In summary, existing deep learning methods make the keyphrase generation task in virtue of designing some additional loss functions. However, the generation accuracy and the exposure bias problem still exist in the current deep learning methods. In this paper, RegRL-KG provides a framework on these developments for further improvements.

2.2 Reinforcement learning

Many works have applied reinforced algorithms to assist the Seq2Seq model in the keyphrase generation task [4, 24]. Considering that the number of generated keyphrases is fewer than that of target keyphrases, a reinforcement learning method [4] with adaptive rewards under the guidance of evaluation scores is proposed to generate both sufficient and accurate keyphrases. Closely to our approach, the authors employ a reinforcement algorithm with adaptive rewards to directly optimize the tasks’ evaluation metrics, such as F1@5 and F@M metric scores. These metrics provide a reasonable way to measure the quality of the generated keyphrases. However, the model that utilized the prediction results as rewards requires a valuable work of post-processing and thus generates large training variance. Furthermore, the limited exploration of their algorithm also undermines the model’s accuracy to the quality of predicted keyphrases. Recently, there has been a renewed push in the use of reinforcement learning algorithms to offer alternatives for keyphrase generation. Swaminathan et al. [24] use a generative adversarial network (GAN) to learn the rewards for the generator. Nevertheless, a GAN network is unstable and difficult to train [21]. More importantly, the trade-off between exploration and generation is a key problem for reinforcement learning models. To avoid training instability, we adopt an actor-critic based reinforcement learning with a precisely designed reward to ensure a more stable and accurate generation environment.

3. Background

In this section, we introduce Markov Decision Process (MDP) [23] and the actor-critic algorithm.

3.1 Markov decision process

A standard discounted Markov decision process can be formalized as $M=(\mathcal{O},\mathcal{S},\mathcal{A},P,R,\rho_{0},\gamma)$ with environment observation $\mathcal{O}$ , state space $\mathcal{S}$ , action space $\mathcal{A}$ , state transition function $P(s^{\prime}\mid s,a)$ , reward function $R(s,a)$ , the distribution $\rho_{0}$ of the initial state $s_{0}$ , and discount factor $\gamma\in(0,1)$ . The goal of the agent is to find a policy $\pi(a\mid s)$ to maximize the cumulative discounted reward $R_{\pi}(s)$ . $R_{\pi}(s)$ denotes the expected reward at state $s$ when following policy $\pi:R_{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot\mid s)}[R(s,a)]$ . At each time step $t$ , the agent receives a state $s_{t}$ and maps it to an action using policy $\pi$ . After rolling out $\pi$ , the agent receives a scalar reward $r_{t}$ and moves to the next state $s_{t+1}$ . The process continues until the agent reaches a terminal state of an episode. A standard reinforcement learning method sets the target of return maximization as the minimization of the loss function $\mathcal{L}_{\text{RL}}$ of the RL agent.

3.2 Actor-critic algorithm

Deep deterministic policy gradient (DDPG) is a model-free RL algorithm developed for working with continuous high dimensional actions spaces [25]. DDPG uses an actor-critic architecture maintaining a deterministic policy (actor) $\pi:\mathcal{S}\rightarrow\mathcal{A}$ and an action-value function approximation (critic) $Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ . Actor-critic training starts with calculating the advantage $A_{\pi}(s,a)$ of taking an action $a$ in the state $s$ , defined as the value $Q_{\pi}(s,a)$ of taking an action $a$ compared to the average value $V_{\pi}(s)$ of taking all possible valid actions in that state $s$ . The action-value function $Q_{\pi}(s,a)$ describes the expected return from state $s$ after taking action $a$ and subsequently following policy $\pi$ . In the AC framework with a GAE, the state-value function $V_{\pi}(s)$ , action-value function $Q_{\pi}(s,a)$ and the advantage function $A_{\pi}(s,a)$ are simplified as follows:

$\displaystyle V_{\pi}(s)=\sum_{t=0}^{\infty}\gamma^{t}\mathbb{E}_{s_{t}\sim P_% {\pi}^{t}(s)}[R_{\pi}(s_{t})]$ (1) $\displaystyle Q_{\pi}(s,a)=R(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim P(s^{\prime% }\mid s,a)}[V_{\pi}(s^{\prime})]$ (2) $\displaystyle A_{\pi}(s,a)=Q_{\pi}(s,a)-V_{\pi}(s)$ (3)

where $P_{\pi}^{t}(s)$ denotes the distribution over state space after taking $\pi$ for $t$ steps starting from state $s$ . The advantage function is commonly used to reduce variance compared to using the returns directly. $V_{\pi}(s)$ is predicted by a critic network and $R(s_{t})$ is the reward received at step $t$ . The critic network looks back and estimates the sum of the expected rewards using the value function $V_{\pi}(s)$ , which also provides important guidance on the training of model. Specifically, the action decoder or actor is updated according to the actor loss:

$\displaystyle\mathcal{L}_{\text{actor}}=\mathbb{E}_{\pi}[\log\pi(a^{*}\mid% \mathrm{s})A_{\pi}(s,a)]$ (4)

$a^{*}$ is the target policy in the training phase. The actor $\pi(a^{*}\mid\mathrm{s})$ is trained using the policy gradient, which is based on the fact that each step $t$ in the action decoding process is conditioned on all the previously decoded portions. The critic’s job is to approximate the actor’s action-value function $Q_{\pi}(s,a)$ , prompting the value of the critic’s prediction approximate to its true underlying value. The critic is updated by minimizing the loss function:

$\displaystyle\mathcal{L}_{\text{critic}}=\frac{1}{2}\sum_{t}({Q}_{\pi}(s_{t},a% _{t})-V_{\pi}(s_{t}))^{2}$ (5)

Additionally, a regularization term $H(\pi)$ that maximizes the entropy of the policy is utilized to encourage the agent to explore more diverse target words as well as prevent the agent from prematurely converging on a trajectory.

$\displaystyle\mathcal{L}_{\text{Entropy}}(s_{t},a_{t})=\sum_{a\in\pi(a\mid% \mathrm{s})}P(a\mid s_{t})\log P(a\mid s_{t})$ (6)

4. Methodology

This section introduces an L1 regularized reinforcement learning method, called RegRL-KG, to construct a Markov Decision Process for the keyphrase generation task and applies the actor-critic algorithm with L1 policy regularization to improve the Seq2Seq model. In Section 4.1, the keyphrase generation problem is first formalized. Then the RL method is presented in Section 4.2, the implementation details of the train and inference phases are given in Section 4.3, followed by the joint training loss in Section 4.4.

Figure 1.

Architecture overview for RegRL-KG. The left part of the figure is for deep learning part which includes the actor network and the right is for the reinforcement learning and L1 policy regularization. The blue lines are for the exploration phase. The red lines are for the generation. The green lines are for the critic network.

4.1 Problem definition

Our task is keyphrase generation. Assuming a source text $\displaystyle\bm{X}=\{x_{1},x_{2},\ldots,x_{M}\}$ with $M$ tokens, our objective is to predict the target keyphrases $\displaystyle\bm{Y}=\{y_{1},y_{2},\ldots,y_{N}\}$ , with $N$ keyphrases. $T$ is the maximum length of $N$ keyphrases.

4.2 Overview of the RegRL-KG

The overall pipeline of our method is illustrated in Fig. 1. we adopt a model-based reinforcement learning approach for keyphrase generation. Our framework can be extended to reinforcement learning based on the existing deep learning model. The networks consist of Observation $O$ , State $S$ , an RL agent (an actor and a critic network) and an exposure bias evaluation part. In the generation phase during training, we learn all of the parameters of these networks. Given the source text $\bm{X}$ and target keyphrases $\bm{Y}$ , the individual feature $\bm{E}_{X}$ is encoded by a Transformer encoder and $\bm{E}_{Y}$ is embedded by an Embedding Layer. $\bm{E}_{X}$ and $\bm{E}_{Y}$ are then passed through the actor to generate the target keyword $\bm{Y}$ . The actor computes each keyword logit $\bm{k}_{\text{logit}}$ from both present and absent probability distributions. The role of the critic network is evaluate the generability of the actor. In summary, we treat a KG session as a trajectory and formulate a KG episode as a markov decision process, where an RL agent is designed to manipulate the final generation policy. To train this network efficiently and effectively, several strategies are assisted in the training process. First, the actor-critic network is introduced to drive the final softmax policy towards regions of the state-action space with higher Q-values and improves its ability to discover a better policy. Second, a related reward is developed to ensure exploration stability during the reinforcement process. Additionally, a exposure bias reduction objective through a L1 policy regularization loss is proposed to guarantee convergence when confronted large deviation. More formal definitions and module details are given below. Algorithm 1 provides the pseudocode of the RegRL-KG algorithm using actor-critic as its policy gradient component.

[h] : RegRL-KG for Keyphrase Generation[1] Training the source text and the ground-truth keyphrase pairs ( $\bm{X}$ , $\bm{Y}$ );An Actor $\pi(a\mid s)$ and Critic $Q(a,s)$ Keyphrase probability $p$ Initialize the exploration buffer $B$ . all observations $O$ in $\mathcal{O}$ % Exploration phase during training $N$ trajectories $T$ keywords the actor predicts policy $\pi(O)$ and the critic computes value $v$ take the probability and action tuple $(p,a)$ from policy $\pi(O)$ receive reward $R$ and compute the advantage $A$ change to next $O^{\prime}$ % Generation phase during training all samples in buffer $B$ agent predicts new policy $\pi^{\prime}(O)$ and value $v^{\prime}$ greedy predicts action $a^{*}$ compute MLE loss $\mathcal{L}_{\mathrm{M}\mathrm{LE}}=-\sum_{t=0}^{T}\log p(w_{t}\mid w_{0:t-1},% \bm{X})$ compute policy regularization loss $\mathcal{L}_{\text{Bias}}=\sum\left\|\bm{k}_{\text{logit}}^{e}-\bm{k}_{\text{% logit}}^{g}\right\|_{1}$

compute the RL loss $\mathcal{L}_{\mathrm{RL}}=\beta_{1}\mathcal{L}_{\text{actor }}+\beta_{2}% \mathcal{L}_{\text{critic }}-\beta_{3}\mathcal{L}_{\text{Entropy }}$ compute the total loss $\mathcal{L}=\mathcal{L}_{\mathrm{MLE}}+\alpha_{1}\mathcal{L}_{\mathrm{RL}}+% \alpha_{2}\mathcal{L}_{\mathrm{Bias}}$ Update the actor network using the policy gradient: $\nabla\mathcal{L}_{\text{actor}}\sim\mathbb{E}_{\pi}[\nabla\log\pi(a^{*}\mid% \mathrm{s})A_{\pi}(s,a)]$ Update critic network with TD gradient: $\nabla\mathcal{L}_{\text{critic}}\sim\nabla\frac{1}{2}\sum_{t}(Q_{\pi}(s_{t},a% _{t})-V_{\pi}(s_{t}))^{2}$ backpropagate the combined loss update the exploration Buffer $B$

4.2.1 Observation, state and transition functions

The KG episode is modeled as a Markov Decision Process (MDP) defined by state space $\mathcal{S}$ , transition function $P(s_{t+1}\mid s_{t},a_{t})$ and action space $\mathcal{A}$ . The model starts with the transformation of raw source text $\bm{X}$ and target keyphrases $\bm{Y}$ into embedding vectors $\bm{E}_{X}$ and $\bm{E}_{Y}$ . We define the tuple $O=(\bm{X},\bm{Y})$ as the observation. When constructing the MDP, we define the source text and the target keyphrase pair $(\bm{X},\bm{Y}_{1\ldots t-1})$ as the state at each time step $t$ . As all target keyphrases correspond to the same source text, we assume all tasks share the same state space, while each is characterized by a unique combination of transition and reward functions. The execution of the policy results in an “action”, the prediction of the next token. An action is a keyword to be generated, namely $a_{t}=w_{t}$ , in the action space $\mathcal{A}$ . After determining the action, our agent reaches the next state $(\bm{X},\bm{Y}_{1\ldots t})$ . The Markov property is strictly preserved as $(\bm{X},\bm{Y}_{1\ldots t})$ models previous knowledge. When the agent issues an action $a_{t}$ , the environment transitions to state $s_{t+1}$ from the current state $s_{t}$ with probability $P(s_{t+1}\mid s_{t},a_{t})$ . Since teacher forcing is utilized for training in KG, the environment is deterministic and the state transition probability is determined $P(s_{t+1}\mid s_{t},a_{t})\equiv 1$ . After executing all the actions in a keyphrase, the agent updates its internal state. For every source text $\bm{X}$ and a target keyphrase $\bm{Y}$ with $T$ words, when $T$ steps are taken to accomplish a keyphrase generation session, this is referred to as a trajectory.

4.2.2 Reward function

Given a source text, the agent learns to predict a sequence of target keyphrases. The reward function for the agent is usually learned using a relative MLE loss, which provides a measure of similarity between the predicted and the target keyphrases. We want the generation phrase can produce optimal keyphrases with greedy policy, while the exploration phrase can further produce possible better trajectories with stochastic policy. This encourages training of the agent as this would require better actions to follow from the same state vector in the exploration phase. Based on this, we leverage the disparity of the generated probability between the exploration and training stage as a relative reward:

$\displaystyle R=\left\{\begin{array}[]{ll}\mu^{-}&\operatorname{loss}_{\text{% mle}}>\operatorname{loss}_{\text{mle}_{\_}\text{explore}}\\ \mu^{0}&\operatorname{loss}_{\text{mle}}=\operatorname{loss}_{\text{mle}_{\_}% \text{explore}}\\ \mu^{+}&\operatorname{loss}_{\text{mle}}<\operatorname{loss}_{\text{mle}_{\_}% \text{explore}}\end{array}\right.$ (7)

Keywords that reduce MLE loss is rewarded by $\mu^{+}$ , equal case gets a negative penalty $-\mu^{0}$ and increase case are penalized by $-\mu^{-}$ . $\mu^{-}>\mu^{+}$ is set to discourage alternating diverging and converging steps. This design can be interpreted that, the influence of the agent will weaken if the action has larger expected cumulative rewards compared to other actions, and vice versa. Depending on the characteristic, the relative reward can alleviate the possible of divergent actions and hence correct the agent’s stochastic policy. Actor-critic network optimizes with a related reward to ensure training convergence.

4.2.3 Policy optimization

Based on the MDP formulation and the designed reward, we use AC algorithm with GAE along the step of a keyphrase and about the final generation probability distribution as action space. Our goal is to learn a stochastic policy $\pi$ that maximizes the expected cumulative reward for the keyphrase generation actor. Note that the action space $\mathcal{A}$ of our method is the holistic discrete action space $\mathcal{S}$ , which contains all the available words in the dictionary and the out-of-vocabulary (OOV) items in the source text. In practice, it is very hard to make an explicit control on the generated probability distribution. We treat the merged softmax policy layer as an actor network to generate different keywords in each step. The actor learns the probability distribution $p$ to decide action $a$ by the policy $\pi$ from state $\mathcal{S}$ . This decision generation procedure corresponds to policy and action process. The probability distribution-action tuple $(p,a)$ can control the generation’s execution in the deterministic environment. Apart from the actor network, a critic network value head estimates the baseline $V_{\pi}(s)$ based on the combination of state and action. During RL update, the baseline is subtracted from the returns of the actions to compute the advantage. The pre-training provides an unbiased evaluation for each action but generates high variance [23], so we apply temporal difference (TD) gradient to smooth the variance. The advantage function $A_{\pi}(s,a)$ can be seen as instant feedback and measures how much a token is better than the exploration process. If the $A_{\pi}(s,a)$ is high, it means the action is reasonable in a keyphrase generation trajectory, and it is less necessary to correct the output distribution, and vice versa. It helps the RL pay more attention to the generation control. The actor model and the critic model are both neural network-based estimators at each timestep $t$ . The structure of the actor network is implemented as multiple linear layers followed by a softmax operation, shown in the Eq. (8). The critic network is implemented as Eq. (9) by fully-connected layers of size [512, 512, 1], where 1 for the value estimate.

$\displaystyle\pi(a\mid\mathrm{s})=\operatorname{softmax}((\mathrm{\bm{k}_{% \text{logit}}^{g}}\mathbf{W}_{\mathrm{1}}+b_{1})\mathbf{W}_{\mathrm{2}}+b_{2})$ (8) $\displaystyle V_{\pi}(s)=(\mathrm{\bm{k}_{\text{logit}}^{g}}\mathbf{W}_{% \mathrm{3}}+b_{3})\mathbf{W}_{\mathrm{4}}+b_{4}$ (9)

with $\mathbf{W}_{i}$ and $b_{i}(i=1,2,3,4)$ being trainable parameters.

4.2.4 Alleviate exposure bias by policy regularization

Besides the problem discussed above, there still exists the exposure bias problem between the exploration and generation phases. To simultaneously ensure the training stability and convergence, the policy requires to be close to that in the exploration phase after an RL updates. This is achieved by the L1 policy regularization loss. In the keyphrase generation task, the main idea of adding the L1 policy regularization is to regularize the policy during policy optimization. As long as the output of the exploration phase is near the ground truth that embedded in the generation phase, it is regarded as a correct prediction. Therefore, we are more concerned about the target samples in the exploration phase whose prediction is far from the true value. In other words, we hope that after minimizing the exposure disparity, there is a small difference of the probability distribution between the generation and the exploration phases. In conjunction with L1 policy regularization, Teacher forcing and RL objectives, RegRL-KG can be jointly optimized to further limit divergence of the policy and ensure the consistency of policy in the generation phase.

4.3 Training, evaluation and testing

4.3.1 Exploration and generation phases in the Training stage

The trade-off between exploration and generation is a key problem for reinforcement learning models. The training process iterates between the exploration and the generation phases. During exploration, stochastic policy is used to gather intermediate data and is more likely to produce better trajectories. To avoid exposure bias, the exploration model makes the next keyword based on its previous predictions. In the generation phase, teacher forcing, L1 policy regularization, and greedy sampling strategies are adopted to guarantee accuracy. In this manner, exploration and generation complement each other.

4.3.2 Training, evaluation and testing stage

Training stage

In the training stage, we design the exploration and the generation phases. At each step, we calculate $p(y_{t+1}\mid\bm{X},y_{t})$ to obtain the next word $y_{t+1}$ using teacher forcing in the generation phase and simulate the test stage with the previously generated keyword in the exploration phase.

Evaluation and testing stages

In evaluation and the testing stage, after generating a keyword, we sequentially use the previously predicted keyword to generate the final keyword probability. Additionally, just the actor network is used to produce the really keyphrase probability given the source text.

4.4 Training loss

We apply the actor-critic algorithm with an L1 policy regularization based on the MLE loss function $\mathcal{L}_{\text{MAE}}$ to improve the Seq2Seq model. The overall loss of our framework consists of the RL loss $\mathcal{L}_{\text{RL}}$ , the policy regularization $\mathcal{L}_{\text{Bias}}$ , and the mainstream loss $\mathcal{L}_{\text{MLE}}$ with the balancing hyperparameters $\alpha_{1}$ and $\alpha_{2}$ :

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{MLE}}+\alpha_{1}\mathcal{L}_{\text% {RL}}+\alpha_{2}\mathcal{L}_{\text{Bias}}$ (10)

The details of each loss are summarized as follows:

MLE loss

Traditionally, deep learning-based methods employ maximum likelihood estimation with the teacher-forcing technique [9, 30], which maximizes the loglikelihood by lowering the cross-entropy loss during the training stage. Based on MLE, the KG task is to maximize the conditional possibility given the source text $\bm{X}$ :

$\displaystyle\mathcal{L}_{\text{MLE}}=-\sum_{t=0}^{T}\log p(w_{t}\mid w_{0:t-1% },\bm{X})$ (11)

where $w_{t}$ is the target keyword at the time step $t$ , $w_{0:t-1}$ is the previously generated words.

Policy regularization loss

We impose an L1 policy regularization loss to directly minimize the prediction logit of the exploration $\bm{k}_{\text{logit}}^{e}$ and the generation $\bm{k}_{\text{logit}}^{g}$ phases in the training process:

$\displaystyle\mathcal{L}_{\text{Bias}}=\sum\|\bm{k}_{\text{logit}}^{e}-\bm{k}_% {\text{logit}}^{g}\|_{1}$ (12)

RL loss

${\mathcal{L}_{RL}}$ are represented as the concatenation of the Eqs (4)–(6).

$\displaystyle\mathcal{L}_{\text{RL}}=\beta_{1}\mathcal{L}_{\text{actor}}+\beta% _{2}\mathcal{L}_{\text{critic}}-\beta_{3}\mathcal{L}_{\text{Entropy}}$ (13)

where $\beta_{i}(i={1,2,3})$ is the weight for each component part in RL.

5. Experimental implementation

We introduce the benchmark datasets and baselines, evaluation metrics and the corresponding experimental settings for the experiments.

5.1 Datasets

Following [18], we train the model on the KP20k training dataset and evaluate on the KP20k test dataset. The trained model is also evaluated by four scientific article datasets, including Inspec [11], NUS [20], Krapivin [16] and SemEval [14] that has different data characteristics from the source training dataset. Each sample from these datasets consists of a title, an abstract and some keyphrases. The released KP20k dataset consists of 567,830 high-quality scientific articles from various computer science domains, where 527,830 for training, 20,000 for validation, and 20,000 for testing. In the training dataset, the title and abstract are concatenated as a source text. The text pre-processing steps are operated on both source texts and ground-truth keyphrases, including tokenization, lowercasing and replacing all digits with symbol $<$ digit $>$ . In addition, examples with empty ground-truth keyphrases are removed. After the cleanup, the final KP20k dataset contains 509,818 articles for training, 20,000 for validation, and 20,000 for testing. Ground truth keyphrases are sorted by their order of first occurrence in the source text. Keyphrases that do not appear in the source text are appended to the end. For a fair comparison, we use the released pre-processed dataset as previous works [18, 4, 6, 24, 31]. The dataset statistics are shown in Table 1.

Table 1
Statistics of the five scientific article testing datasets

Dataset	#Samples	Avg.#KP	Avg. $\|$ KP $\|$	% Absent
Inspec	500	9.79	2.48	26.42
NUS	211	10.81	2.22	45.36
Krapivin	400	5.83	2.21	44.33
SemEval	100	14.43	2.38	55.61
KP20k	20,000	5.26	2.04	37.23

Avg. #KP: the average number of keyphrases. Avg. $|$ KP $|$ : the average length of keyphrase. % Absent: the average ratio of the absent keyphrases.

5.2 Baselines

We compare four state-of-the-art baselines that apply a deep learning model to the keyphrase generation task. We also experiment with two RL training methods catSeqTG-2RF1 [4] and GANMR [24].

catSeq [32]. The first baseline model is a RNN-based Seq2Seq model with copy mechanism.

catSeqTG [7]. An extension of catSeq with additional title encoding and cross-attention.

catSeqTG-2RF1 [4]. An extension of catSeqTG applied the RL-based finetuning algorithm with adaptive rewards to improve the performance of the Seq2Seq model.

GANMR [24]. Another extension of catSeq using a discriminator to produce rewards for RL.

ExHiRD-h [6]. A method based on catSeq using an exclusive hierarchical decoding framework with a hard exclusion mechanism to improve generation performance.

Transformer [31]. A Transformer-based Seq2Seq model with the copy and attention mechanism to achieve the keyphrase generation task.

SETTRANS [31]. The up-to-date Transformer-based Seq2Seq model with additional control codes and a K-step target assignment mechanism via bipartite matching to generate keyphrases.

5.3 Evaluation metrics

In line with prior works [31, 6], we evaluate both present and absent keyphrase predictions using a metric with a variable cutoff macro-average F1@M and a metric with a fixed cutoff macro-average F1@5. F1@M calculates the total generated keyphrases with the ground-truth keyphrases. For the metric F1@5, if the generated keyphrase number is less than five, we randomly append incorrect keyphrases until it obtains five predictions. Porter Stemmer is adopted before calculating whether two keyphrases are matched.

5.4 Implementation details

We implement our framework on the Transformers toolkit1 and py-Torch 1.1.0.2 We train our framework on one Tesla V100 GPU with 32GB memory. The optimization uses Adam [15] with an initial learning rate 0.0001, and halves the learning rate if the perplexity on the validation set stops decreasing for one epoch. Early stopping is employed during training. Batch size is set to 24. For the reward function, we experimentally determine $(\mu^{+},\mu^{0},\mu^{-})=(0.5,-0.1,-0.6)$ to guide the agent. The discount factor is set as $\gamma=0.50$ and the GAE factor is $\lambda=0.95$ . The value of $\beta_{i}(i={1,2,3})$ is set as $(0.5,1.0,0.1)$ . The RL loss term is scaled by $\alpha_{1}=0.01$ and the policy regularization loss is set to $\alpha_{2}=1$ . Experiments are repeated using three different random seeds and the averaged results are reported.

6. Results and analysis

Present and absent predictions are conducted to demonstrate the effectiveness of the generality of the approach in Section 6.1. For the illustration of the RL training, the RL analysis trained on the KP20k training datasets is shown in Section 6.2. Section 6.3 provides a case study for the generated keyphrases. In Section 6.4, an ablation study is evaluated on KP20k testing dataset and shows how each component influences the model.

Table 2
Present keyphrases prediction results of all models. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.289 ${}_{3}$ indicates 0.289 $\pm$ 0.003)

Models	Inspec		NUS		Krapivin		SemEval		KP20k
	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M
catSeq	0.225	0.262	0.323	0.397	0.269	0.354	0.242	0.283	0.291	0.367
catSeqTG	0.229	0.270	0.325	0.393	0.282	0.366	0.246	0.29	0.292	0.366
catSeqTG-2RF1	0.253	0.301	0.375	0.433	0.3	0.369	0.287	0.329	0.321	0.386
GANMR	0.258	0.299	0.348	0.417	0.288	0.369	–	–	0.303	0.378
ExHiRD-h	0.253	0.291	–	–	0.286	0.347	0.284	0.335	0.311	0.374
Transformer	0.281 ${}_{5}$	0.325 ${}_{6}$	0.370 ${}_{7}$	0.419 ${}_{10}$	0.315 ${}_{8}$	0.365 ${}_{5}$	0.287 ${}_{14}$	0.325 ${}_{15}$	0.332 ${}_{1}$	0.377 ${}_{1}$
SETTRANS	0.285 ${}_{3}$	0.324 ${}_{3}$	0.406 ${}_{12}$	0.450 ${}_{7}$	0.326 ${}_{12}$	0.364 ${}_{12}$	$\textbf{0.331}_{\textbf{20}}$	0.357 ${}_{13}$	0.358 ${}_{5}$	0.392 ${}_{4}$
KGAgent	$\textbf{0.289}_{\textbf{3}}$	$\textbf{0.328}_{\textbf{1}}$	$\textbf{0.420}_{\textbf{5}}$	$\textbf{0.459}_{\textbf{5}}$	$\textbf{0.341}_{\textbf{3}}$	$\textbf{0.384}_{\textbf{3}}$	0.326 ${}_{6}$	$\textbf{0.358}_{\textbf{7}}$	$\textbf{0.367}_{\textbf{0}}$	$\textbf{0.403}_{\textbf{1}}$

6.1 Present and absent keyphrase predictions

6.1.1 Comparison with the state-of-the-art algorithms

The present and absent results are reported in Tables 2 and 3 for clear comparisons with RegRL-KG, respectively. Among the seven existing state-of-the-art baselines, our method performs best in both present and absent metrics on almost all of the datasets. As shown in Tables 2 and 3, RegRL-KG obtains 0.90%/1.10% present F1@5 and F1@M improvement over the best baseline (SETTRANS) on the KP20k dataset as well as a present F1@5/F1@M improvement of 0.40%/0.40% (Inspec), 1.40%/0.90% (NUS), 1.50%/2.00% (Krapivin) and $-$ 0.50%/0.10% (SemEval) respectively. As a whole, our method RegRL-KG achieves an exceeding improvement over the up-to-date method SETRANS on the average present metrics and obtains outstanding performance on the average absent metrics. We also can see that the variance of the RL method shows smaller fluctuation on most of the datasets. Considering that the number of all possible keywords in the action space is very large and the difficulty of our keyphrase generation problem is particularly high, these results suggest that our RL agent performs more effectively in regard to control the generation ability. The performance of RegRL-KG also shows that metrics manage to obtain more increase when adding effective exploration searching compared with the deep learning based-methods. Our approach focuses more on RL optimization, which acts complementary to the above-mentioned supervised methods. The experiments demonstrate that RL can successfully build a present as well as an absent policy, and RegRL-KG can achieve more efficient, impressive and heuristic grounding results. This convincingly shows the great potential of the policy control for keyphrase generation.

Table 3
Absent keyphrases prediction results of all models. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.020 ${}_{1}$ indicates 0.020 $\pm$ 0.001)

Models	Inspec		NUS		Krapivin		SemEval		KP20k
	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M
catSeq	0.004	0.008	0.016	0.028	0.018	0.036	0.016	0.028	0.015	0.032
catSeqTG	0.005	0.011	0.011	0.018	0.018	0.034	0.011	0.018	0.015	0.032
catSeqTG-2RF1	0.012	0.021	0.019	0.031	0.030	0.053	0.021	0.030	0.027	0.050
GANMR	0.013	0.019	0.026	0.038	0.042	0.057	–	–	0.032	0.045
ExHiRD-h	0.011	0.022	–	–	0.022	0.043	0.017	0.025	0.016	0.032
Transformer	0.010 ${}_{2}$	0.019 ${}_{4}$	0.028 ${}_{2}$	0.048 ${}_{2}$	0.032 ${}_{1}$	0.060 ${}_{4}$	0.020 ${}_{5}$	0.023 ${}_{3}$	0.023 ${}_{1}$	0.046 ${}_{1}$
SETTRANS	$\textbf{0.021}_{\textbf{1}}$	$\textbf{0.034}_{\textbf{3}}$	0.042 ${}_{2}$	0.060 ${}_{4}$	0.047 ${}_{7}$	0.073 ${}_{11}$	0.026 ${}_{3}$	0.034 ${}_{5}$	0.036 ${}_{2}$	0.058 ${}_{3}$
RegRL-KG	0.020 ${}_{1}$	0.030 ${}_{2}$	$\textbf{0.043}_{\textbf{3}}$	$\textbf{0.063}_{\textbf{4}}$	$\textbf{0.048}_{\textbf{3}}$	$\textbf{0.074}_{\textbf{7}}$	$\textbf{0.030}_{\textbf{1}}$	$\textbf{0.039}_{\textbf{2}}$	$\textbf{0.041}_{\textbf{0}}$	$\textbf{0.064}_{\textbf{0}}$

6.1.2 Limited improvement on Inspec and SemEval datasets

As indicated in Tables 2 and 3, the results we obtain are close to those of SETTRANS on the present SemEval and even declines in the absent Inspec metrics on average. By our observation, the reason might be the shorter average keywords in the training dataset. The average number of each keyphrase in Inspec/SemEval is about 9.79/14.43, which is longer than 5.26 average number in KP20k dataset. This implies that there is less exploring scope that can not serve as supportive evidence when generating the longer keyphrases. The shorter keyphrase number may cause partial observation and the narrower scope will induce the actor with poor generative ability. We believe a better exploration (in our case, increasing the length of trajectories) strategy that can broaden the scope of observation may help the agent to master such harder or unseen dataset.

6.2 Reinforcement learning analysis

Figure 2.

Reward and return (upper left), value and advantage (upper right), logits for exploration and training (lower left), and entropy for exploration and source (lower right).

In Fig. 2, we show the metric change of RegRL-KG from the perspective of reinforced learning, using KP20k training dataset as a case.

6.2.1 Reward, return, value and advantage

Reward and return

We can observe that the reward metric of RegRL-KG in Fig. 2a is consistent with the score of return metric. The RegRL-KG approximates the expected cumulative rewards and thus provides accurate returns. The reward and return scores grow with the training length and become relatively steady after about 2500 batches. Despite a little fluctuation from batch 27000 to 30000, AC and L1 policy regularization can pull the model back to a stationary state. The sharp oscillation is because RL randomly selects tokens to explore the action space at the exploration phase, which makes the training of policy difficult. As batch increases, the reward judgment aligns well with the results of return scope, indicating the controlled policy works effectively with the environment and state. RegRL-KG can interact well with the reward and the accumulated return, which also implies that RegRL-KG can learn a reasonable policy strategy.

Value and advantage

Figure 2b presents the intermediate process of the reinforcement learning. Both the value and advantage scores are smooth and consistent when training. We evaluate the RegRL-KG by measuring 1) if it reduces the temporal difference error. 2) if it approximates the expected value. As evidenced in Fig. 2b, the change of advantage is initially quite drastic but quickly becomes much smoother with the number of iterations and the convergence is consistent with the value metric training. When the exploration expands, we find the TD errors become smoother while the approximation difference increases. It is because as the exploration scale becomes larger, the model facilitates the TD learning while tries to reduce the training variance. The steady performance of the training advantage curve reveals that it is also viable to use TD advantage policy gradients in AC. In addition, the value curve of critic network is increasing during the training stage, demonstrating the effectiveness of state embedding. We guess the key reason is that the critic of the RL agent already knows the values of certain states and actions. Therefore, it can be used to select which action should be followed in a principled and globally consistent manner.

6.2.2 The exploration is beneficial for the generability

We investigate the influences of exploration by analyzing the logits (Fig. 2c) and the entropy (Fig. 2d) between the exploration and the training phases. We find the keyword logit under the training phase is a little higher than that of exploration, which is generally consistent with the entropy results. Without exploration, the improvement made by AC itself is very limited, which is smaller than that with exploration, because it is difficult for AC to learn a complex word-level critic without exploration. As the training converges, the keyword logit and the entropy learning curves in Fig. 2c and d further demonstrate the ability of exploration. The exploration phase forces the model to explore different words at key positions.

6.3 Case study

In order to get a better understanding of RL control and policy regularization, we follow the ExHiRD-h method and provide the same sample with the generated keyphrases on the KP2k testing dataset. From Fig. 3, RegRL-KG generates more accurate keyphrases comparing to the best baseline SETTRANS. It can be seen that RegRL-KG directly highlights the present keyphrases “co-verification” and “debugging” as well as the the absent keyphrases “computer software”. For the content of prediction, RegRL-KG and ExHiRD-h both rank first and predict more accurate keyphrases than the latest method SETTRANS. Nevertheless, the number of the predicted keyphrases by RegRL-KG is six, which is more closer to the target keyphrase number four when compared to the keyphrase number nine by ExHiRD-h. RegRL-KG can improve the number of unique keyphrases by a large margin, which is extremely beneficial from the combination of RL control and L1 policy regularization. Correspondingly, RegRL-KG solves the overgeneration problem that existed in the baselines. Beside, less repeated keyphrases are generated through the RL agent and the policy regularization. For instance, ExHiRD-h produces the keyphrase “computer software” at two times. However, RegRL-KG generates it just one time, which demonstrates that our proposed method is more powerful in avoiding duplicated keyphrases. The same lower average duplication keywords appeared in the Table 4 also quantitatively confirm this phenomenon, which indicates RegRL-KG arranges the keyword probability distributions in a more precise and compact way.

Figure 3.

An example of generated keyphrases by baselines and RegRL-KG on the KP20k testing dataset. The correct predictions are bold and the present keyphrases are underlined. The digit in parentheses denotes the duplication number of the predicted keyphrases (e.g., “co-verification (1)” represents the keyphrase “co-verification” is generated one time).

6.4 Ablation study

To understand what makes our framework effective, we conduct an ablation study on the SemEval dataset. We ablate RL control and L1 policy regularization loss. The effect of individual component is further reported in Table 4. SETTRANS [31] is used as the baseline.

6.4.1 Full model

From Table 4, we can conclude that utilizing actor-critic and L1 policy regularization strategy simultaneously can learn a strong policy. After adding RL control and L1 policy regularization, the scores of many predicted keyphrases are improved over the baselines. It indicates a positive relationship between the RL control (as well as L1 policy regularization) and the ground-truth, which is consistent with the motivation in the introduction section. Beside, the average present keyphrase number is closest to the ground-truth and the predicted number for absent ranks second among all the baselines. The average duplication of the predicted keyphrases is lowest. Moreover, SETTRANS is built on Transformers, so our improvement is not relied on the replacement of any language model, but using actor-critic algorithm and L1 policy regularization. The comprehensive metrics show that the combination of reinforcement control and L1 policy regularization is superior to each component.

6.4.2 RL control

The final performance is heavily decreased compared to the original full model and is even worse than the baseline SETTRANS. We believe that our AC network designed for maintaining the deep learning structure allows the model to stably generate keyphrases and leads the performance improvement as shown in Table 4. We find that RL plays a crucial role. After adding the RL network, RegRL-KG leads to a 0.50%/1.00% improvement in the F1@5/F1@M metric. This means that the actor is a good explorer for achieving proper actions conditioned on the distributional probability in the training phase. In addition, it is noticeable that, the AC setting is more stable across training. It reveals that, by using the actor-critic algorithm, we can drastically improve the model’s performance and make RL more stable across training due to the stationarity of the environment.

Table 4
Ablation study. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.367 ${}_{0}$ indicates 0.367 $\pm$ 0.000)

Models	Present			Absent			Dup
	$\textit{F1@5}(\uparrow)$	$\textit{F1@M}(\uparrow)$	$\#\textit{PK}$	$\textit{F1@5}(\uparrow)$	$\textit{F1@M}(\uparrow)$	$\#\textit{AK}$	$\textit{Dup}(\downarrow)$
Oracle	–	–	3.31	–	–	1.95	–
SETTRANS	0.358 ${}_{5}$	0.392 ${}_{4}$	5.10	0.036 ${}_{2}$	0.058 ${}_{3}$	2.01	0.080
SETTRANS $+$ RL	0.363 ${}_{0}$	0.402 ${}_{1}$	5.03 ${}_{4}$	0.040 ${}_{0}$	0.063 ${}_{0}$	2.12 ${}_{3}$	0.070 ${}_{2}$
SETTRANS $+$ L1	0.366 ${}_{1}$	0.402 ${}_{1}$	5.05 ${}_{7}$	0.039 ${}_{0}$	0.062 ${}_{1}$	2.09 ${}_{8}$	0.068 ${}_{1}$
RegRL-KG	$\textbf{0.367}_{\textbf{0}}$	$\textbf{0.403}_{\textbf{1}}$	$\textbf{4.95}_{\textbf{2}}$	$\textbf{0.041}_{\textbf{0}}$	$\textbf{0.064}_{\textbf{0}}$	2.04 ${}_{2}$	$\textbf{0.067}_{\textbf{2}}$

Avg. #PK: the average number of present keyphrases. Avg. #AK: the average number of absent keyphrases. Dup: the average duplication of the predicted keyphrases. The lower duplication ratio is better. Oracle: the gold average keyphrase number. The closest values to the oracles are bold.

6.4.3 L1 policy regularization

Adding the only L1 policy regularization leads to a higher F1@5 but a higher F1@M metric score when compared to the baseline SETTRANS. Unfortunately, the full model RegRL-KG reaches highest on both high F1@5 and F1@M metric scores, and obtains the highest task success. This is due to the fact that L1 policy regularization limits the exploration and the generation degree, which learns the generation manner and enables the explored action in the exploration phase be analogously reflected in the target training phase.

7. Conclusion

Deep reinforcement learning methods have offered an intuitive paradigm for exploring goal driven keyphrase generation task. The sheer size of the generation action space, however, has proven to be out of the reach of existing algorithms. In this paper, we introduced RegRL-KG, a novel learning agent that demonstrates the feasibility of scaling reinforcement learning towards keyphrase generation actions spaces with hundreds of millions of actions. The key insight to being able to efficiently explore such large space is the combination of actor-critic algorithm and L1 policy regularization control. The actor-critic network serves as a means for the agent to accumulate information about the target distribution probability while the L1 policy regularization is adopted to ensure model convergence and training stability. Together they constrain the vast space of possible actions into the compact space of sensible ones. The RegRL-KG framework demonstrates that RL algorithms can work well with the keyphrase generation task when combined with DNNs methods. The generality of our approach is evaluated on five datasets and our approach achieves the state-of-the-art for comparable methods.

From a reinforcement learning perspective, RegRL-KG can be viewed as a form of ‘goal-driven guide’ that promotes exploration towards states with higher long-term returns, boosts accurate of explored policies and introduces stability for policy control. The principal mechanism behind RegRL-KG is the capability to incorporate both modes of learning: learning directly from the high accuracy of deep learning methods while being aligned to maximize the long-term returns by leveraging the controllable RL optimization. In this paper, we use a standard AC to operate the backbone deep learning methods on the holistic distribution. Incorporating decoupling sub-distributions, e.g., the copy and generating probability distributions, is an exciting area of future work.

Footnotes

https://huggingface.co/transformers/.

https://pytorch.org/.

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by the National Natural Science Foundation of China under Grant 62272100, the Consulting Project of Chinese Academy of Engineering under Grant 2023-XY-09, and in part by the Major Project of the National Social Science Fund of China under Grant 21ZD11 and the Fundamental Research Funds for the Central Universities.

References

Barto

A.G.

Sutton

R.S.

and Anderson

C.W.

, Neuron like adaptive elements that can solve difficult learning control problems, IEEE Transactions on Systems, Man, and Cybernetics SMC-13(5) (1983), 834–846.

Boudin

Gallina

and Aizawa

, Keyphrase generation for scientific document retrieval, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20’, Online, 2016, pp. 1118–1126.

Bousmalis

Trigeorgis

Silberman

Krishnan

and Erhan

, Domain Separation Networks, in: Proceedings of the Advances in Neural Information Processing Systems, NIPS 16’, Barcelona, Spain, 2016, pp. 343–351.

Chan

H.P.

Chen

Wang

and King

, Neural keyphrase generation via reinforcement learning with adaptive rewards, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 19’, Florence, Italy, 2019, pp. 2163–2174.

Chen

Chan

H.P.

P.J.

Bing

L.D.

and King

, An integrated approach for keyphrase generation via exploring the power of retrieval and extraction, in: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 19’, Minneapolis, Minnesota, 2019, pp. 2846–2856.

Chen

Chan

H.P.

P.J.

and King

, Exclusive hierarchical decoding for deep keyphrase generation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20’, Online, 2020, pp. 1095–1105.

Chen

Gao

Y.F.

Zhang

J.N.

King

and Lyu

M.R.

, Title-guided encoding for keyphrase generation, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 19’, Honolulu, Hawaii, USA, 2019, pp. 6268–6275.

Chiu

Sahu

S.K.

Thomas

Sengupta

and Mahdy

, Autoencoding keyword correlation graph for document clustering, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20’, Online, 2020, pp. 3974–3981.

Cho

Merriënboer

B.V.

Gulcehre

Bahdanau

Bougares

Schwenk

and Bengio

, Learning phrase representations using RNN encoderâ€“decoder for statistical machine translation, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 14’, Doha, Qatar, 2014, pp. 1724–1734.

10.

Gollapalli

S.D.

X.L.

and Yang

, Incorporating expert knowledge into keyphrase extraction, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 2017’, San Francisco, California, USA, 2017, 31(1).

11.

Hulth

, Improved automatic keyword extraction given more linguistic knowledge, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 03’, 2003, pp. 216–223.

12.

Hulth

and Megyesi

B.B.

, A study on automatically extracted keywords in text categorization, in: Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics, ACL 06’, Sydney, Australia, 2006, pp. 537–544.

13.

Jiang

Jin

Duan

Z.Y.

and Zhang

C.S.

, RL-duet: Online music accompaniment generation using deep reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 20’, 2020, pp. 710–718.

14.

Kim

S.N.

Medelyan

Kan

M.Y.

and Baldwin

, SemEval-2010 task 5: Automatic keyphrase extraction from scientific articles, in: Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval 10’, Uppsala, Sweden, 2010, pp. 21–26.

15.

Kingma

D.P.

and Ba

, Adam: A method for stochastic optimization, in: Proceedings of the International Conference on Learning Representations, ICLR 15’, San Diego, USA, 2015.

16.

Krapivin

and Marchese

, Large dataset for keyphrase extraction, Technical report, University of Trento, 2009.

17.

Luan

Ostendorf

and Hajishirzi

, Scientific information extraction with semi-supervised neural tagging, in: Proceedings of the Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 17’, Copenhagen, Denmark, 2017, pp. 2641–2651.

18.

Meng

Zhao

S.Q.

Han

S.G.

D.Q.

Brusilovsky

and Chi

, Deep keyphrase generation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 17’, Vancouver, Canada, 2017, pp. 582–592.

19.

Samy

Oriol

Navdeep

and Noam

, Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks, in: Proceedings of the 28th International Conference on Neural Information Processing Systems – Volume 1, Montreal, Canada, 2015, pp. 1171–1179.

20.

Nguyen

T.D.

and Kan

M.Y.

, Keyphrase extraction in scientific publications, In Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, pages 317–326, 2007.

21.

Sarmad

Lee

H.J.

and Kim

Y.M.

, Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 19’, Long Beach, Canada, 2019, pp. 5898–5907.

22.

Schulman

Moritz

Levine

Jordan

and Abbeel

, High-dimensional continuous control using generalized advantage estimation, in: Proceedings of the International Conference on Learning Representations, ICLR 16’, San Juan, Puerto Rico, 2016.

23.

Sutton

R.S.

and Barto

A.G.

, Reinforcement learning: An introduction, IEEE Transactions on Neural Networks 9(5) (1998), 1054–1054.

24.

Swaminathan

Zhang

H.M.

Mahata

Gosangi

Shah

R.R.

and Stent

, A preliminary exploration of GANs for keyphrase generation, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 20’, Online, 2020, pp. 8021–8030.

25.

Lillicrap

T.P.

Hunt

J.J.

Pritzel

Heess

Erez

Tassa

Silver

and Wierstra

, Continuous control with deep reinforcement learning, in: Proceedings of the International Conference on Learning Representations, ICLR 16’, San Juan, Puerto Rico, 2016.

26.

Z.P.

Z.D.

Liu

X.H.

and Li

, Modeling coverage for neural machine translation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 16’, Berlin, Germany, 2016, pp. 76–85.

27.

Ushiku

Mori

Kameko

and Tsuruoka

, Game state retrieval with keyword queries, in: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 17’, Shinjuku, Tokyo, Japan, 2017, pp. 877–880.

28.

Wang

and Cardie

, Domain-independent abstract generation for focused meeting summarization, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 13’, Sofia, Bulgaria, 2013, pp. 1395–1405.

29.

Wang

Chan

H.P.

King

Lyu

M.R.

and Shi

S.M.

, Topic-Aware Neural Keyphrase Generation for Social Media Language, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 19’, Florence, Italy, 2019, pp. 2516–2526.

30.

Williams

R.J.

and Zipser

, A learning algorithm for continually running fully recurrent neural networks, Neural Computation 1(2) (1989), 270–280.

31.

J.C.

Gui

Luo

Y.C.

Y.G.

and Zhang

, One2Set: Generating diverse keyphrases as a set, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 21’, Online, 2021, pp. 4598–4608.

32.

Yuan

X.D.

Wang

Meng

Thaker

Brusilovsky

D.Q.

and Adam Trischler, One size does not fit all: Generating and evaluating variable number of keyphrases, in Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20’, Online, 2020, pp. 7961–7975.

33.

Zhang

Wang

Gong

Y.Y.

and Huang

X.J.

, Keyphrase extraction using deep recurrent neural networks on Twitter, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 16’, Austin, Texas, 2016, pp. 836–845.

34.

J.W.

Michel

Chris

Gao

J.F.

and Bill

, A Diversity-Promoting Objective Function for Neural Conversation Models, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, 2016, pp. 110–119.

RegRL-KG: Learning an L1 regularized reinforcement agent for keyphrase generation

Abstract

Keywords

1. Introduction

2. Related work

2.1 Keyphrase extraction and generation

2.2 Reinforcement learning

3. Background

3.1 Markov decision process

3.2 Actor-critic algorithm

4.2 Overview of the RegRL-KG

4.2.1 Observation, state and transition functions

4.2.2 Reward function

4.3 Training, evaluation and testing

4.3.1 Exploration and generation phases in the Training stage

4.3.2 Training, evaluation and testing stage

Training stage

Evaluation and testing stages

4.4 Training loss

MLE loss

Policy regularization loss

RL loss

5.1 Datasets

Table 1 Statistics of the five scientific article testing datasets

5.3 Evaluation metrics

5.4 Implementation details

6. Results and analysis

Table 2 Present keyphrases prediction results of all models. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.289 3 indicates 0.289 ± 0.003)

6.1.1 Comparison with the state-of-the-art algorithms

Table 3 Absent keyphrases prediction results of all models. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.020 1 indicates 0.020 ± 0.001)

6.2 Reinforcement learning analysis

Reward and return

Value and advantage

6.2.2 The exploration is beneficial for the generability

6.3 Case study

6.4.1 Full model

6.4.2 RL control

Table 4 Ablation study. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.367 0 indicates 0.367 ± 0.000)

7. Conclusion

Footnotes

Acknowledgments

References

Table 1
Statistics of the five scientific article testing datasets

Table 2
Present keyphrases prediction results of all models. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.289 ${}_{3}$ indicates 0.289 $\pm$ 0.003)

Table 3
Absent keyphrases prediction results of all models. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.020 ${}_{1}$ indicates 0.020 $\pm$ 0.001)

Table 4
Ablation study. The best results are bold. The subscript represents the corresponding standard deviation (e.g., 0.367 ${}_{0}$ indicates 0.367 $\pm$ 0.000)