Personalized scientific and technological literature resources recommendation based on deep learning

2.1 Collaborative filtering-based methods

The CF-based methods mainly focus on the explicit feedbacks (e.g., scores, comments, tags) or implicit feedbacks (e.g., views, downloads) of the papers of other users with similar research interests, and then recommends papers to the users who have research interests similar to the current users [11, 13]. Bogers et al. [14] apply the traditional CF method to academic papers recommendation from CiteULike database, and find that the user-based recommendation method is superior to the item-based method due to data distribution. Parra et al. [15] compare three user-based CF algorithms and find that increasing the number of ratings in the traditional collaborative filtering recommendation algorithm can reduce the uncertainty caused by insufficient ratings. Fernando et al. [16] extract the research topics and cardinality, which represents the importance of the topic in the paper, from the titles, abstracts, keywords and other key information of papers. And the method converts the typical tuples: {user, paper, ratings} into a set of tuples:{ paper, topic, cardinality}. Finally, on the basis of the converted tuples, the approach utilizes CF recommendation to achieve personalized recommendations for papers and topics.

Usually, the CF methods rely on the inner product of latent vectors to model the interaction of users and papers. But such product is not efficient enough to capture the complex structure of user interaction data [8]. Besides, CF methods often suffer from the problem of cold-start, new users and newly published papers often do not have sufficient ratings, and therefore they cannot be recommended.

2.2 Content-based filtering methods

The CBF methods use researcher’s feature data to construct researchers’ profiles and extracts paper information to generate paper representation. Then the methods calculate and compare the similarities between the researcher profiles and the paper models, and then recommend papers that have features similar to users’ profiles [17]. In essence, the CBF approaches are dependent on manually-designed features. As academic papers tend to have a wide spectrum of auxiliary information such as authors, titles, abstracts, keywords, and journals, the CBF methods are widely used in academic paper recommendation [2]. Sugiyama [18] proposes a generic model that recommends scholarly papers relevant to a researcher’s interests by capturing their research interests through their past publications. Kazemi et al. [19] compare two well-known content-based paper recommendation algorithms, namely TF-IDF (term frequency–inverse document frequency)-based and word embedding-based. Experimental results show that the method based on embedding representation has achieved a better performance. Kanakia et al. [20] propose a mixed recommendation model based on content and co-citation, and apply the model to Microsoft academic which contains more than 200 million English papers and patents. However, the linear similarity measurement of the CBF methods cannot fully capture potential interaction features between users and papers. Zhelezniak et al. [21] hold that cosine similarity is essentially equivalent to Pearson correlation coefficient, and they find that Pearson correlation is not always suitable for similarity measurement, proving that cosine similarity is not so effective for all word vectors.

2.3 Graph-based methods

The GB recommendation methods focus on building graph networks that typically demonstrate how the academic entities are connected. The entities in such networks (e.g., researchers, papers, venues, topics) are different nodes of the graphs, and their mutual interaction relationships (e.g., publishing, citing, viewing, downloading) serve as the edges of the graphs [3, 22]. Hence, the paper recommendation problem is transformed into a graph search problem; related papers are recommended to users by calculating the similarity between user nodes and paper nodes [23, 24]. The core of the graph-based recommendation methods lies in the construction of the graphs, which can be composed of co-citation [25], co-author [26], and social network [27]. Pan et al. [28] construct a heterogeneous graph to represent citations and content information in the papers, and use a similarity learning algorithm based on the heterogeneous graph for paper recommendation. Ma et al. [29] study the recommendation of newly published papers in heterogeneous paper networks, and propose a new method based on a meta-graph recommendation model. However, the previous research has not consider the user’s research interests and the contents of the papers, therefore they cannot well satisfy users’ personalized needs.

2.4 Deep learning-based methods

In recent years, deep learning is gaining an increasing popularity in a wide range of recommendation systems (RS). Many studies suggest that integrating deep learning with RS can significantly boost the performance [9, 10]. Covington et al. [30] propose a deep CF learning based model for YouTube’s video recommendation. Users’ watch history, search history, and geographic information are utilized as features to train the neural networks, and then recommendation lists are generated. He et al. [8] propose a generic framework based on a neural network, namely Neural network-based Collaborative Filtering (NCF), which uses multi-layer perceptron to learn the interaction features between users and items. The experiments with two real-world public datasets (MovieLens, Pinterest) confirm the method is superior to the existing CF methods. Xue et al. [31] propose a deep item-based CF approach (DeepICF) based on the NCF framework, and demonstrates that DNNs have great potential to model the higher-order and non-linear interactions between items by using the similar methods as in Ref. [8]

Currently, deep learning-based personalized recommendation methods are fully employed in the fields of movies, books, music, news, social networks, applications, e-commerce, and blogs [32]. Nevertheless, limited attention has been paid to recommend academic papers based on deep learning. Bhagavatula et al. [33] propose a citation recommendation method; a query document is firstly embedded into a vector space, then candidates are generated through the nearest neighbor method, finally the candidates are ranked using the trained neural network model. This method has not taken into account the user’s research interests to support personalized recommendation. Hassan [34] proposes a paper recommendation model based on Recurrent Neural Networks (RNNs). As the author claims, this method can better discover continuous and latent semantic features of the papers, thereby improving the paper recommendation quality. Unfortunately, there is no evidence to validly support such claim. Wang [35] proposes a hybrid RS that combines CF and CBF. In the CBF model, word embedding learned by neural networks is utilized to map users and papers to the word vector space to calculate the similarity. However, the latent interactions between users and papers cannot be captured by this method.

2.5 Literature summary

In general, personalized recommendation is of great significance to solve the problem of information overload that pervasively exists in scientific and technological literature resources services. Many attempts have been made on developing personalized academic papers recommendation methods. However, there are still some noticeable drawbacks of the reported methods. Firstly, the traditional methods like CF and CBF rely on linear combination of the same weight between the elements of two vectors to obtain the relevance of researchers’ attributes and papers’ attributes. However, it is difficult to represent the latent high-order interaction characteristics between users and items. Secondly, although several academics [33, 35] have utilized deep learning to recommend papers, but they mainly use deep learning to model auxiliary information or to learn word embedding. Moreover, these methods still calculate the interaction features between users and papers based on simple linear similarities between user vectors and paper vectors.

In order to address the extant knowledge gaps in the papers recommendation methods, a deep learning-based personalized recommendation method, namely Deep Content-based Collaborative Filtering (DeepCCF), is proposed for scientific and technological literature resource recommendation. The method combines content-based method (CBF) [20] and neural network-based collaborative filtering method (NCF) [8]. Firstly, the problem of scientific and technological paper recommendation is transformed into a binary classification task. Secondly, word2vec is utilized to learn the word embedding of the corpus and to build the academic paper topic model based on Term Frequency–Inverse Document Frequency (TF-IDF) and word embedding. Thirdly, the researchers’ search and view history, and published papers are utilized to model the research interests of the researchers. Deep neural networks (DNN) are then used to learn the nonlinear and complicated high-order feature interactions between users’ research interests and academic paper topic models. Finally, the top k recommendation list is generated based on the outputs of the DNN model. Eventually, it can provide researchers with personalized recommendation services for literature resources most related to their research interests.

The overall structure of the DeepCCF RS is shown in Fig. 1. The system is comprised of two functions: (1) feature engineering and (2) model training and recommendation generation. In the feature engineering, the word2vector tool is used to train word embedding, and the academic paper embedding is constructed through the word embedding and TF-IDF for each paper. The users’ search history is transformed into a set of feature words by preprocessing such as word segmentation, which is necessary for filtering stop words in Chinese papers. Then, the users’ search history model is constructed by word embedding. The users’ view history model is built according to users’ view records and paper embedding. Similarly, the users’ published papers model is built according to users’ published papers and paper embedding. The model of users’ research interests is constructed by the users’ search history model, view history model as well as published papers model.

OPEN IN VIEWER

Fig. 1

Deep Content-based Collaborative Filtering framework.

During model training, the model of users’ research interests and paper embedding are used as input features, users’ view records are used as labels, the concatenation vectors of the users’ research interests embedding and paper embedding are fed into deep neural networks to train the interaction function. During recommendation generation, the top K recommendation list is generated by calculating the output probability ranking corresponding to the input users’ research interests embedding and the candidate papers embedding.

Since it might be difficult to get the full texts of papers due license and access limitations, we select the title and abstract of the paper to construct the paper embedding. The title of the paper summarizes the main content of the paper with concise sentences, and the abstract is a brief statement of the whole paper, including the goals, methods and performances. Such parts are more informative than the rest sectors of the paper. Therefore, the title and abstract can represent the main content of the paper.

The procedure of paper embedding generation is shown in Fig. 2. First, the word embedding of the title and abstract of all papers in the academic paper library are generated, the weights of the feature words are then calculated, and the linear combination of the word vectors weighted by the feature word weights is used to generate the paper vector representation.

OPEN IN VIEWER

Fig. 2

Modeling process of paper embedding.

First, the word2vec is used to learn the word embedding. Word embedding, also known as low-dimensional distributed word representation, plays an important role in the field of natural language processing. Word2vec is a famous tool created and published by Tomas Mikolov at Google to produce word embedding. Typical, the approach consists of two models: the continuous bag-of-words model (CBOW) and the continuous skip-gram model [36, 37]. Word2vec takes as its input a large corpus of text and produces a vector space through training three-layer neural networks, with each unique word in the corpus being assigned a corresponding vector in the space. Several approaches such as GloVe, [38] word2vecf [39] have been proposed to create word embedding after word2vec. Especially, in recent years, a lot of research works focus on the Chinese word embedding such as CWE, GWE, JWE, cw2vec, ALWE are proposed to address the word embedding challenge in Chinese texts, and most of them are essentially based on word2vec [40]. Nevertheless, since there is no more significant advantage than word2vec which is of simplicity and efficiency, and considering that word embedding is not our core research in this work, skip gram model is selected to learn the word embedding and take other methods aside as future work. The corpus for training the model are titles and abstracts from all the papers.

TF-IDF is used to form the paper topic model [41]. It is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. The weight of a word in a document is proportional to the frequency of its appearance, while the specificity of a word can be quantified as an inverse function of the number of documents in which it occurs. The general calculation formula of TF-IDF is as follows. TFIDF = tf ij · idf j = tf ij · log ( N n j )(2) where tf _ij refers to the number of the word w _j appears in the paper p _i . N is the total number of papers in the academic papers library. n _j is the number of papers where the feature word w _j has occurred.

A paper embedding is generated by linear combination of words embedding occurring in the paper title and abstract weighted by the TF-IDF value. In addition, considering that the content in the title can better reflect the topic of the article, we set the weight of the words occurring in the title as 3, and the weight of the words occurring in the abstract as 1 [42]. The low-dimensional vector representation of the paper is finally generated as follows.

M paper = 3 ∑ w ɛ title TFIDF ( w ) · w ˆ + ∑ w ɛ abstract TFIDF ( w ) · w ˆ 3 ∑ w ɛ title TFIDF ( w ) + ∑ w ɛ abstract TFIDF ( w )(3)

where TFIDF (w) refers to the TFIDF value of the word w, w ˆ refers to the embedding of word w.

In order to recommend papers related to users’ current research interests, users’ search history, view history and published papers are chosen as the key features of the model of users’ research interests, the reasons of this chance are shown as follows.

It is a knowledge-driven process for researchers to search literature through academic search engine, which is usually based on a specific research purpose and domain knowledge to determine one or several keywords for retrieval, therefore the searching records can reflect the researchers’ research fields and interests.

As shown in Fig. 3 of users’ research interests modeling process, the search history embedding is calculated by the average values of words embedding, and the view history embedding and published paper embedding are built by average of paper embedding. Finally, the users’ research interests embedding is generated by the weighted average of searching history embedding, viewing history embedding and published papers embedding.

OPEN IN VIEWER

Fig. 3

Structure of users’ research interests embedding.

(1) Users ’ search history embedding generation

In this paper, the users’ recent search history is processed into a set of search words. Based on the characteristic word vector trained in Section 3.2, the words that do not exist in the corpus can be eliminated. The dense search history vector which generalizes the retrieval history is calculated by average of search words vectors. M search = 1 n ∑ * 20 c w i ∈ V w i ∈ W w ˆ i(4) where w _i refers to the i-th word, V refers to the corpus, W is the user search word set, and w ˆ i is the word embedding of w _i , n refers to the size of the search queries. In this paper, we select the users’ interaction records for the last 60 days to calculate the search history vector.

(2) Users ’ view history embedding generation

We utilize the users’ view records in the past period of time, and generate the researchers’ viewing history vector by calculating the average value based on the paper embedding. M view = 1 n ∑ i = 1 n P i(5) where n refers to the number of papers viewed by the user, and P _i is the vector of the i-th paper viewed by the users. In this paper, we select the users’ interaction records for the last 60 days to calculate the searching history vector.

(3) Users ’ published papers embedding generation

Considering the time characteristics of papers, generally speaking, researchers’ research interests change with time; the later the papers are published, the more representative of their current research interests can be. Thus, we first rank the papers according to the time points of publication, and then generate the embedding of researchers’ published papers through calculating the weighted average of the paper embedding [43]. M papers = ∑ i = 1 n ( e - γ d n → i P i ) ∑ i = 1 n e - γ d n → i(6) where n is the number of papers published by the researcher, e^-γd_n→i is the weight of each paper, and d_n→i is the year difference between the i-th paper and the n-th paper, γ is the time decay factor, and (0 ⩽ γ ⩽ 1), in this paper, γ is initialized to 0.2, and P _i is the vector of the i-th paper.

(4) Users ’ research interests embedding generation

In order to keep the dimension size of user vector unchanged, we build the users’ research interests embedding by calculating the weighted average of search history embedding (Equation (4)), view history embedding (Equation (5)) and published papers embedding (Equation (6)).

M user = α search M search + α view M view + α papers M papers α search + α view + α papers(7)

where α _search , α _view , α _papers denote to the weight of searching history embedding, viewing history embedding and published papers embedding respectively, and their values are all initialized to be 1.

After obtaining the paper embedding and user embedding, the reported recommendation methods directly calculate cosine similarity to measure the similarity between them, and then recommend the literature resources to users according to the similarity ranking [3, 44]. In this paper, the user embedding and paper embedding are connected as the input layer, and the standard MLP (Multi-Layer Perception) is used to learn the interaction function between the users and papers. The model architecture is shown in Fig. 4. The features of the model are the users’ research interests embedding M _user and paper embedding M _paper . In this paper, the logistic sigmoid function is used as the activation function of the output layer to limit the prediction results to the interval [0, 1], and it is modeled as the conditional probability. Since the number of scientific and technological literature resources is much too larger than the number of researchers, bringing about the problem of data sparsity. Thus, we use Rectified Linear Units (ReLU) as the activation function of the hidden layers of neural networks. Cross entropy loss function serves as the loss function of the model. The tower structure is used to build the neural network, that is, the number of higher-level neurons is reduced by half compared to the previous layer [45]. During the training, we use Lecun normal distribution initializer to initialize the weight and bias of each neuron. And Adam optimizer is used to calculate and update the model parameters.

OPEN IN VIEWER

Fig. 4

Deep neural network model.

The framework of the proposed personalized scientific and technological literature resources recommendation system proposed is shown in Fig. 5. First of all, users search and browse literature through academic paper platform, in which users’ interaction records are stored. During the model training process, all the data such as the title and abstract of the papers in the database are exported to train the word embedding and then to construct the paper embedding. The users’ viewing records, search records, published papers and other data are exported to construct the users’ research interests embedding. The user vector and the paper vector are connected to feed into the DNN model together with the view records as labels for offline training. Since the papers’ corpus are rather enormous, given a user embedding, it will take an extra-long time to find the most relevant papers directly through DNN model. It is difficult to get the recommendation list in real time. Therefore, we divide the personalized recommendation of academic papers into two stages: recalling and ranking; these are common processes in the domain of RS [30]. In the recall phase, various recommendation methods are used to quickly obtain the papers related to the users’ research interests as the candidate set. Then, the candidate set is input into the DNN model to filter out the papers irrelevant to the users’ research interests and thereby recommend the corresponding resources to the user.

OPEN IN VIEWER

Fig. 5

Method of personalized scientific papers recommendation based deep learning.

During the generation of candidates, the user embedding is updated in real time according to users’ interaction records. For each user, we use the Nearest Neighbor Search to find the n paper vectors with the highest similarity to the user embedding as candidate sets. Commonly used (approximate) nearest neighbor retrieval methods are the K-Nearest Neighbor (KNN) [46] and Locality-Sensitive Hashing (LSH) [47]. In order to reduce the computational complexity of finding similar vectors, we first use the spherical k-means method [20] to divide the paper sets into multiple subsets. In this sense, we only need to calculate the similarity between the user vector and the centroid of each subset. To seek out the cluster with the highest similarity and to determine a certain number of papers as the candidate list, we follow the Ref. [20] to initialize the centroid count k = 80,000, which is generated with the assistance of the MAG topic stamping algorithm. For sure, we can also use a variety of recommended methods to generate candidate list to improve the overall recall rate, such as user-based CF method, paper-based collaborative filtering method, co-citation-based method, and co-occurrence-based method.

After acquiring the candidate list, we use the deep learning model to sort out the top-K papers most related to the users’ research reinterests as the recommended list. The specific implementation steps are as follows:

Concatenate the user vector and the paper vector in the candidate list to input DNNs model for prediction, and sort the prediction results by value in descending order.

With the publication of new papers and the increase of user interaction data, the model needs to be updated in real time. Firstly, hundreds of new papers are added to the resource database every day, and the word embedding, paper embedding and clustering model need to be updated regularly to ensure that the newly published papers can also be presented to users. Secondly, with the continuous accumulation of platform data, the recommendation model also needs to be trained and updated constantly to grant the recommendation results with increasing accuracy.

4.1 Dataset

Public datasets have been released for paper recommendation research and are used for recommendation research. Typical examples include CiteSeer’s dataset that contains public academic corpus, citation maps, author data and author collaboration networks [48], and Docear’s dataset that comprises of paper metadata, citation networks, user inforation, user created mind map and paper recommendation information [49]. In addition, Aminer has created several datasets including academic social networks, research topics, knowledge maps and other sorts of data 1 . However, these datasets are only accessible for some specific recommendation methods like Co-citation, CF and CBF. In addition, there is no open academic paper data set including users’ view records, and search query tokens. Another point worth noting is that the previous studies mainly focus on the recommendation of English papers. In fact, there are many distinctive differences in papers of different languages. One most notable difference is that English papers are based on words with spaces, while Chinese papers are based on sentences that consist of words without space. Therefore, in feature engineering, specific preprocessing such as word segmentation is needed. In this experimental sector, we focus on Chinese papers.

All the experimental data in this work is acquired from Wanfang Data, which is the leading scientific and technological literature resources service provider in China 2 . By May 2020, there are more than 130 million journal papers in Wanfang corpus, most of which are Chinese papers. Users typically conduct multiple searches when they visit the Wanfang Knowledge Service Platform. Once finding papers that they think may be relevant to their researches, they would click on the papers to browse the detail pages, they may also browse the detail page of references or citations of current papers. Wanfang stores user interaction records, such as searching queries, viewed paper IDs, and time stamps, into the database with the users’ consent. In general, based on the author information contained in the metadata of the papers, such as the author’s name and institution, Wanfang generates a unique author ID and attribute papers with the same author information to that author automatically. To address the potential errors in author identification, the users on Wanfang can manage their publications manually, using orders like claiming the authorship of the papers and disproving the possible authorship.

Due to the limited computing power, we do not use all users and papers in Wanfang corpus for experiments. Instead, we design a simple sampling process: Firstly, we use the paper query API provided by Wanfang to get the list papers through a query. In this article we choose “artificial intelligence ()” as the query keyword. After removing all the duplicate data, English papers and other invalid data, we use them as the paper set of this experiment. We also acquire the list of authors of these papers. Secondly, by using the user query API, we get the authors who have certified on Wanfang. Thirdly, we use the APIs of user view records, search records, and published papers to obtain detailed information about authors. Finally, we get these authors’ information including their viewing history, searching history and published papers from Wanfang, and such collected behavior records are from September 6 to November 6, 2020. Then we add the viewed papers and published papers into the paper set and processed again. The information of the experimental dataset is shown in Table 1.

During the generation of training data, users’ viewing records (M _user , M _paper ) are selected as positive samples. M _user refers to the user embeddings when the user browses the paper. It should be noted that M _user is constantly updated with the users’ interaction. M _paper refers to the paper embedding, and M _paper will also change as the corpus is updated. For each positive sample, we use two types of negative samples:

Papers that the given user has not viewed: Given M _user , randomly select papers that the user has not viewed as the negative samples (M _user , M paper - ).

We adopt the leave-one-out method to evaluate the performance of the deep learning model [8]. For each user, we chose the latest view record as the test set and use the remaining data for training set. The total number of training datasets reaches 1,189,280. For each positive example, we randomly select four negative instances. For each test sample, 99 papers are randomly selected from the corpus that the user had not viewed as negative samples. Without changing the order of the test samples, we feed the data into the DNNs model in batches to calculate the predicted ranking of the 100 test samples, and evaluate the model performance via analyzing the ranking of the positive samples.

For comparison purpose, we choose the content-based method (CBF) based on embedding and NCF method (NeuMF) [8] as baseline methods:

The CBF method. The representation of papers is formed by word embedding and TF-IDF. The users’ searching history vector is constructed based on his or her searching records according to Equation 4. Similarly, the view history vector is constructed with view records according to Equation 5, and the published paper vector is constructed based on published paper records according to Equation 6. The users’ vector is constructed based on searching history vector, query history vector and published paper vector according to Equation 7. Recommended paper list is generated by computing cosine similarity of candidate paper vector and user vector. The effectiveness of this method has been verified by the previous research [19, 20]. We follow the setting of [20] to report the best performance.

The NCF method. The identity of a user and a paper is used as the input feature to generate a dense vector through an embedding layer. The user embedding and paper embedding are then fed into a multi-layer neural architecture and a GMF layer separately, the last hidden layer of MLP layers and GMF layer are concatenated to generate the NeuMF layer. We follow the setting of [8] to report the best performance.

In this experiment, Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) are employed to evaluate the performance of the ranking list. HR is widely used to measure the recall rate. Given a real test set, HR can measure the number of times the papers in the test set appear in the top K recommendation list. HR @ k = Number of Hits @ k | GT |(8) where the numerator refers to the number of times the papers in the test set appear in the top k recommendation list. It is set as a hit if the papers in the test set appear in the top k recommendation list. The denominator refers to the size of the real test set.

NDCG can reflect the accuracy of recommended results. NDCG takes the correlation of each recommendation result as the score of the recommendation list by weighted summation. The higher the weight of the items in the top of the list, the greater the ranking effect of the papers in the test set in the TOPK recommendation list. The higher the value of NDCG, the higher the correlation between the recommendation result and the users’ research interests. NDCG @ k = Z k ∑ i = 1 k 2 rel i - 1 log 2 ( i + 1 )(9) where k refers to the size of the recommendation list, and Z _k is the normalization coefficient, therefore the value of NDCG is 1 under the best recommendation result (i.e., the recommendation list is ranked from high to low correlation). rel _i indicates the relevance of the recommendation result at position i. In this experiment, we use 0 and 1 to represent the relevance; If the paper corresponding to this position is in the test set, rel _i  = 1, otherwise it is 0.

This experiment mainly focuses on Chinese papers recommendation based on deep learning. Instead of training the word embedding with paper corpus, we straightly use the pre-trained Chinese word vectors provided by the Ref. [50] to form paper embedding and user embedding. The word vectors are trained with six large-scale Chinese corpora including Chinese Wikipedia, Baidu-baike (an online Chinese encyclopedia), Zhihu (Chinese social QA platform like Quora), People’s Daily news, Sogou News and Financial News and the word embedding size is 300. In addition, the word segmentation tool named Jieba is used for word segmentation and keyword extraction 3 . Table 2 shows the hyper-parameter settings for deep neural networks. We test the batch size of [100, 500, 1000] and attain best results when the batch size is 500. In addition, we test the learning rate of [0.0001, 0.0005, 0.001, 0.005, 0.01] and attain best results when the learning rate is 0.005.

Since the last hidden layer of the model determines the model’s capabilities, we denote the last hidden layer as predictive factors [8]. DeepCCF-N means that the DeepCCF model has N hidden layers. For example, (Factor = 8, DeepCCF-4) indicates that the hidden layer structure of the neural network is 64⟶32⟶16⟶8, and similar notations for others.

Figure 6 shows the performance of the top-K recommended lists where the ranking position K ranges from 1 to 10. As we can see from the figure, when K = 1, the values of HR and NDCG (0.247) of DeepCCF are slightly lower than CBF (0.256) but higher than NCF (0.225). In addition, the evaluation value of DeepCCF in HR and NDCG are significantly higher than those of the CBF method and NCF method. Especially when K = 10, the values of HR@10 and NDCG@10 of DeepCCF are increased by 26.8% and 18.0% respectively compared to those of CBF and increased by 5.2% and 6.8% respectively compared to the outcomes of NCF. It can be seen that our proposed paper recommendation method has achieved better results than the baseline methods in terms of the evaluation indicators such as recall rate and accuracy. The high recall rate indicates that the method in this paper can obtain more relevant papers, high accuracy means that the recommendation results are more accurate, which is helpful to improve users’ satisfaction. The experimental results indicate that the deep neural networks can well capture the latent interaction characteristics between users and papers, and achieve better performance than the methods based on linear similarity [20].

OPEN IN VIEWER

Fig. 6

Performance of DeepCCF model (Factor = 64,DeepCCF-2.).

In order to evaluate the contribution of individual feature of users’ profiles and to investigate whether the proposed method is effective in solving the cold start problem, we compared the performance of the proposed method with the baseline methods by using different combinations of features to generate user vectors. As shown in Fig. 7, since the NCF method directly embeds the user id and the paper id, the computation process is independent of the feature combinations, so the values of HR and NDCG of the NCF method always remain the same. When the combinations are “viewing history + published papers", “searching history + published papers” and “viewing history + searching history + published papers", our proposed method outperforms CBF and NCF both for HR and NDCG, especially when the combination is “viewing history + searching history + published papers” our method achieves the best performance. While the NCF method achieves the best performance for the combination of “viewing history + searching history” and the CBF method performs the worst consistently. When we review the dataset, we find that there exist at least 200 users who have even no searching/viewing history. This explains why our proposed approach underperforms NCF when the users’ vector is constructed with view history and search history, as it means that many of the users’ vectors are empty when generating training data. Thus, we can conclude that users’ published papers might be a good solution to the problem of recommendation for users without any search or view histories. In addition, the combination of “viewing history + published papers” slightly outperforms that of “searching history + published papers", indicating that users’ view history contributes more to the recall rate and accuracy of recommendation than users’ search history does. Similarly, we can conclude that the ranking order of individual features’ contribution to user profile is the published papers, viewing history and searching history.

OPEN IN VIEWER

Fig. 7

Performance of DeepCCF model (Factor = 64,DeepCCF-2.) with different combinations of features.

We further investigate the effectiveness of the DeepCCF model with different hidden layers. The experimental results are provided in Table 3, where the maximum values of the same predictive factors are marked in bold. When the predictive factors are 8, 16, 32, the values of HR@10 and NDCG@10 of the model both gradually rise as the number of hidden layers increases, which suggests that by adding more non-linear hidden layers, DNNs model can better capture the high-order relationship between users and papers, and boost a better performance. When the value of the prediction factor is too large (64, 128), if we continue to increase the number of hidden layers, overfitting will occur and the performance would be significantly deteriorated. In this experiment, with the other hyper parameters unchanged, the model achieves the best results when the hidden layer’s structure is 128⟶64.

5.1 Contribution and user interface

In this work, a novel solution for personalized scientific and technological literature resources recommendation based on deep learning named DeepCCF has been presented. We integrate neural collaborative filtering method, the implicit feedback of user view history is used to replace user ratings as labels, which effectively solves the problems of lack of explicit data. We combine embedding-based and content-based recommendation method; we utilize word embedding and TF-IDF to form low-dimensional distributed representation of paper topic model, and effectively solving the problem of paper cold start. Based on our approach, even newly published papers can be accurately recommended. We use users’ information such as searching records, browsing records, and published papers to acquire the hidden features of the users’ research interests, effectively solving the problem of user cold start. Even a newly registered user can get a personalized recommended list via our method. We use deep neural networks to learn the latent and nonlinear high-order features between the users’ research interests and the paper topics. Experimental results demonstrate the effectiveness of our approach. We provide a comprehensive solution for personalized scientific and technological literature resources recommendation, including the framework of personalized scientific and technological literature resources recommendation system as well as core algorithms based on deep learning, which is of great practical benefit for both providers and users of scientific and technological literature services. For instance, the scientific and technological literature services providers can aid knowledge workers to acquire effective information in a swift manner via the personalized recommendation system.

Figure 8 shows graphical user interface of our proposed academic paper recommendation system. In this system, personalized paper recommendation list can be generated either through single feature or comprehensive features of published papers, search records, view records, collection records.

OPEN IN VIEWER

Fig. 8

Graphical user interface of paper recommendation system.

In this work, we focus on the personalized recommendation method of academic papers related to users’ research interests. To capture the latent relations between researchers’ research interests and the paper topics, we combine the deep learning technique with traditional recommendation methods like CF and CBF to acquire the deep characterization of papers and the implicit feedbacks between researchers and papers. Admittedly, our work still suffers several limitations, which are listed as follows.

The experimental results suggest that our method achieve a better performance than the selected baseline methods. It is worth noting that there are few standard baselines to compare among many recommendation methods in the domain of academic paper recommendation. One possible reason for this is that the way to process data of different methods is not constant. Hence, it is difficult for us to prove whether our method is the most promising. We will try to address this issue in future works.

Aspects other than recall rate and accuracy have not been considered in this paper, such as quality and popularity. Generally speaking, higher-quality or popular papers are more trusted and recognized by users. In the future research, we will take into consideration the quality and popularity of academic papers recommendation.

Linguistic differences have been ignored in this paper. Actually, a cross-language recommendation could be highly desirable for researchers, especially for those whose native language is not English. The key challenge of our method is how to generate representations of papers in different languages. We will pay more attention on building a cross language academic paper recommendation system.