Attribute interaction aware matrix factorization method for recommendation

Abstract

Matrix factorization (MF) models are effective and easy to expand and are widely used in industry, such as rating prediction and item recommendation. The basic MF model is relatively simple. In practical applications, side information such as attributes or implicit feedback is often combined to improve accuracy by modifying the model and optimizing the algorithm. In this paper, we propose an attribute interaction-aware matrix factorization (AIMF) method for recommendation tasks. We partition the original rating matrix into different sub-matrices according to the attribute interactions, train each sub-matrix independently, and merge all the latent vectors to generate the final score. Since the generated sub-matrices vary in size, an adaptive regularization coefficient optimization strategy and an adaptive latent vector dimension optimization strategy are proposed for sub-matrix training, and a variety of latent vector merging methods are put forward. The method AIMF has two advantages. When the original rating matrix is particularly large, the training time complexity of the MF-based model becomes higher and the update cost of the model is also higher. In AIMF, because each sub-matrix is usually much smaller than the original rating matrix, the training time complexity is greatly reduced after using parallel computing technology. Secondly, in AIMF, it is not necessary to modify the matrix factorization model to incorporate attributes and their interactive information into the model to improve the performance. The experimental results on the two classic public datasets MovieLens 1M and MovieLens 100k show that AIMF can not only effectively improve the accuracy of recommendation, but also make full use of parallel computing technology to improve training efficiency without modifying the matrix factorization model.

Keywords

Recommender systems matrix factorization attribute interaction parallel computation

1. Introduction

Matrix factorization is an important method in machine learning and has been widely and effectively applied in the industry such as product recommendation on Amazon.com [1] and movie recommendation on Netflix.com [2]. It has also been continuously concerned in academia, resulting in a lot of related work such as feature learning and optimization algorithms [3, 4]. Although the mathematical representation of MF model is very concise, it is suitable for processing large-scale data, and the results have good interpretability. Moreover, it is easy to fuse a variety of side information and regularization to improve its accuracy [5, 6, 7, 8].

Matrix factorization (MF) is widely used for rating prediction and recommendation. Given a user-item matrix (also called rating matrix), as shown in the lower right section of Fig. 1 where a large number of rating data is unknown. The main idea is to factorize the original user-item matrix into the user matrix and item matrix. Each vector in the user/item matrix represents the behavior features of a user/item. By multiplying the user matrix with the item matrix we can obtain a new user-item matrix called predicted matrix, which fills the missing data in the original matrix.

Figure 1.

An example of a rating prediction ecosystem. The lower right section is a sparse rating matrix where each ? denotes an unknown rating, the lower left section in blue background shows the attributes of users, the upper right section in green background shows the attributes of items, and the goal of rating prediction is to uncover these unknown ratings and recommend the items with high rating values to related users, which is shown in the upper left section.

The traditional MF algorithm only uses the rating data in the lower right section of Fig. 1 and the attribute information is not involved in the model training process. The fitting relationship between a user and all items is included in a unique latent vector, which can prevent over-fitting to a certain extent but lead to the lack of perception of attribute information.

Attribute awareness is a very important research direction in MF algorithm optimization [5, 7, 8]. The idea is to integrate the attribute information of users or items into the process of factorization by some methods. The related attribute awareness methods we have seen in previous papers can be divided into two categories, modifying the recommendation formula and optimizing the loss function. It should be noted that these methods only obtain attribute information but cannot discover the knowledge of attribute interactions. In Factorization Machines (FMs) model we can see the significant improvements brought about by the interactions between features [9]. Through comparative analysis, we believe that attributes and their interactions can also bring better results for MF model.

To mine the attributes and their interactions, we propose an attribute interaction aware matrix factorization (AIMF) method in this paper by referring to two concepts, interaction introduced in FMs and field introduced in field-aware factorization machines (FFMs) model [10, 11]. We propose to divide the original matrix into sub-matrices, train them independently, and then merge them for the final rating prediction. AIMF solves two issues: First, when the original rating matrix is very large, the time complexity of model training will become very high and the model update cost will also become higher. Generally, the size of the sub-matrix is much smaller than the original matrix, so the training time complexity of the sub-matrix will be greatly reduced. Second, in AIMF, the interaction information of attributes can be incorporated into the model without modifying the matrix factorization model. The main contributions of the paper include the following:

(1)

To the best of our knowledge, this work represents the first step towards absorbing attribute interaction information into matrix factorization learning without modifying the model of matrix factorization. The proposed AIMF method can improve recommendation performance and model training efficiency.

(2)

The proposed AIMF provides insight into which feature interactions contribute more to the prediction at the field level. Two optimization schemes are developed to further improve both the performance of AIFM. A parallel processing technique is used to improve the efficiency of AIMF.

(3)

The experimental results on two well-known datasets show the superiority of the proposed interaction-aware models over the state-of-the-art MF-based methods. The extensive experiments are conducted on two public datasets MovieLens 1M and MovieLens 100K because they contain user and item attribute data while most other public datasets do not contain relevant attribute information. Experimental results show that AIMF outperforms other classic MF based models.

2. Related work

In this section, we mainly introduce the concept of matrix factorization, the related improvement methods, and the analysis of their advantages and disadvantages.

MF models. There are many classic ideas related to MF model. SVD algorithm in linear algebra was originally used to perform matrix factorization operation [6]. Currently, more machine learning-based methods are used to approximate the vacancy value of the rating matrix [12, 13]. MF model is described as follows. There is a $m\times n$ user-item matrix with very high sparsity. The purpose of MF is to factorize the rating matrix into a user matrix and an item matrix by related techniques, which represent the behavioral characteristic of all users and all items. SVD algorithm mainly solves the problem that the user number is not equal to the item number in the user-item matrix. Before it, other algorithms can only solve the case of $m=n$ . However, there are still some issues when SVD algorithm is applied to RSs. To solve the problem of data sparseness, Sharifi et al. proposed a method that first uses the mean value to fill the missing data in the user-item matrix, and then performs factorization operation [14]. This method can alleviate the effect of high sparsity to a certain extent. Koren et al. proposed to use machine learning instead of SVD algorithm to obtain user matrix and item matrix, which can effectively avoid the influence of data sparseness on factorization operation, and the computing time can be greatly reduced [15]. This method, unlike traditional SVD, is effective and more efficient. It won the championship in the competition of Netflix movie recommendation that year. These algorithms ultimately produce a unique user latent vector and a unique item latent vector. Although they can reduce the risk of overfitting, they ignore the impact of user and item attributes, which in turn affects the accuracy of prediction and recommendation.

Attribute aware techniques in MF. The attribute aware techniques are mainly used to solve the cold start problem. As we know, some users have relatively few interactions with the items, like users who just entered the e-commerce shopping website. Due to the lack of rating information of these users, we often adopt some content-based methods to improve the accuracy of recommendations. Some improved methods integrated with attributes are proposed, which are mainly divided into two categories.

(1)
The method of modifying recommendation model. Koren et al. mentioned that by converting attribute information into factor vectors and adding them to the prediction formula, the algorithm can obtain better effect [15]. The predicted score $\hat{r}_{ui}$ can be expressed as

$\displaystyle\hat{r}_{ui}=q_{i}^{T}\left[p_{u}+\sum_{a\in A_{(u)}}y_{a}\right]$ (1)

where $A_{(u)}$ is the attribute set of user $u$ , $\sum_{a\in A_{(u)}}y_{a}$ refers to the sum of the factor vectors of the attributes corresponding to user $u$ . And the predicted score $\hat{r}_{ui}$ incorporating item attribute information can be expressed as

$\displaystyle\hat{r}_{ui}=p_{u}^{T}\left[q_{i}+\sum_{a\in A_{(i)}}X_{a}\right]$ (2)

where $A_{(i)}$ is the attribute set of item $i$ , $\sum_{a\in A_{(i)}}X_{a}$ refers to the sum of the factor vectors of the attributes corresponding to item $i$ .
(2)
The method of modifying loss function. The optimized loss function can be also modified to include the attribute information. Ma et al. proposed a loss function which is expressed as [16]

$\displaystyle\min_{P,Q}L_{2}(P,Q)=\frac{1}{2}\sum_{u=1}^{m}\sum_{i=1}^{n}I_{ui% }(r_{ui}-q_{i}^{T}p_{u}^{2}){}+\frac{\beta}{2}\sum_{u=1}^{m}\sum_{f\in F^{+}(u% )}\textit{Sim}(u,f)\|p_{u}-p_{f}\|^{2}{}+\lambda_{1}\|P\|_{F}^{2}+\lambda_{2}% \|Q\|_{F}^{2}$ (3)

where $I_{ui}$ is an indicator function. It is equal to 1 when $r_{ui}$ is not 0, and equal to 0 when $r_{ui}$ is 0. Actually, $I_{ui}$ is used to avoid the impact of missing data in user-item matrix. $\lambda_{1}\|P\|_{F}^{2}$ and $\lambda_{2}\|Q\|_{F}^{2}$ are the regular items of user and item respectively, $F^{+}(u)$ is the friend list of user $u$ , $\textit{Sim}(u,f)$ is the similarity of $p_{u}$ and $p_{f}$ , $p_{u}$ and $p_{f}$ represent the latent vectors of two users. Ma et al. studied how to integrate the social network into MF algorithms [16]. Although they mainly discussed the integration of social network information in MF, the proposed formula is very general and can also be combined with other additional information [17, 18, 19].

Although these two kinds of methods can mine the attribute information of users and items, they cannot reflect the interaction information of attributes. In this paper, we propose a matrix factorization model, which not only utilizes the user and item attributes in the dataset, but also makes full use of the interaction information between them to improve the accuracy of the prediction. In the model, the attribute pairs mined from attributes and their interactions are used to guide the partitioning of the rating matrix. Each sub-matrix corresponds to a different attribute pair. Each sub-matrix can be independently trained, and their prediction results account for different proportions in the final prediction, which will help improve the efficiency of model learning and the accuracy of prediction, respectively.
3. Proposed method

3.1 Framework

Table 1 shows the description of all the notions to be used in the paper. The formal description of AIMF method is as follows. Suppose $R$ is the rating matrix, $a^{j}$ is an element of the attribute interaction set $A$ , $R^{j}$ is a sub-matrix of $R$ related to $a^{j}$ , and $R^{j}=g(R,a^{j})$ , where $g$ is a partitioning function. Generally, $\lvert R^{j}\rvert\ll\lvert R\rvert$ . We can execute matrix factorization operation on each $R^{j}$ independently. Suppose $p_{u}^{j}$ and $q_{i}^{j}$ represent the user latent vector and the item latent vector related to $R^{j}$ , the predicted rating of $R^{j}$ can be expressed as $\hat{r}_{ui}^{j}=(q_{i}^{j})^{T}p_{u}^{j}$ . Then we can merge all the predicted values as $\hat{r}_{ui}=f(\hat{r}_{ui}^{1},\cdots,\hat{r}_{ui}^{\lvert A\rvert})$ , where $f$ is a merging function. Figure 2 shows the framework of AIMF approach, which consists of three parts.

Table 1
Notation information

Notation	Explanation
$R$	the rating matrix
$r_{ui}$ , $\hat{r}_{ui}$	the rating of user u on item i and the predicted rating
$m, n$	the number of users and items
$A,a^{j},a_{u}^{j},a_{i}^{j}$	attribute interaction set and the $j^{\text{th}}$ element, $a^{j}\in A,a^{j}=(a_{u}^{j},a_{i}^{j})$
$R^{j}$	the $j^{\text{th}}$ sub-matrix of $R$ related to $a^{j},R^{j}\subseteq R$
$r_{ui}^{j},\hat{r}_{ui}^{j}$	an element of $R^{j}$ and the predicted rating of $R^{j}$
$p_{u}^{j},q_{i}^{j}$	the latent vector of user and item related to $R^{j}$
$m^{j},n^{j}$	the number of users and items related to $R^{j}$
$\lambda,\lambda^{j}$	regularization coefficient related to the original MF and the $j^{\text{th}}$ MF
$k,k^{j}$	latent vector length related to the original MF and the $j^{\text{th}}$ MF
$\gamma$	a hyperparameter in MF model training
$e_{ui}^{j}$	$e_{ui}^{j}=r_{ui}^{j}-\hat{r}_{ui}^{j}$
$w_{ui}^{j}$	weight of the $j^{\text{th}}$ latent vector for user $u$ and item $i$
$N_{ui}$	the number of attribute interactions associated with both user $u$ and item $i$
$I(\cdot)$	an indicator function. It returns 1 when the independent variable is not empty, otherwise it returns 0.

Figure 2.

The framework of AIMF approach. It consists of three parts, matrix partitioning, sub-matrices training, and latent vectors merging & rating prediction. The bottom is the original rating dataset.

(1)

Matrix partitioning. The original matrix $R$ is partitioned into multiple sub-matrices such as $R^{j}$ according to attribute interaction $a^{j}$ , i.e., $R^{j}$ is a sub-matrix of $R$ , where the user attribute corresponding to each element is $a_{u}^{j}$ and the item attribute is $a_{i}^{j}$ . The dimensionality of $R^{j}$ is smaller than that of $R$ .

(2)

Sub-matrices training. Each sub-matrix can be trained in parallel, and each matrix training can be optimized in different way. Since the sizes of sub-matrices are different, we will use different regularization coefficients and latent vector lengths to train these sub-matrices. The settings of these two hyperparameters will be explained in detail below.

(3)

Latent vectors merging & rating prediction. Finally, we merge all the latent vectors, which represent different attribute interactions. In this paper, we also propose some latent vector merging methods.

There are three key problems to be solved in this framework. (1) How to get attribute interactions and how to choose attribute interaction set? This is the basis of matrix partitioning. (2) How to perform matrix factorization operations on matrices of different sizes? (3) How to merge the results of these sub-matrices factorization? The following will analyze these several issues.

3.2 Matrix partitioning

The basis for matrix division is attribute interaction, such as age 24 and action movies. The acquisition of the interactive relationship is the premise of the accuracy of AIMF.

First, we preprocess user and item attribute values to obtain an appropriate number and high-quality attribute classification. Generally, the type of attribute value is text, and we use Word2Vec model to extract the characteristics of the attribute value and obtain the word vector. Considering the influence of the number of attribute values, we use Kmeans clustering algorithm to cluster these word vectors, the purpose of which is to select a suitable number of attribute-value categories. The number of attribute value classifications follows a normal distribution, and some of them are very small. At this time, these small number of attribute classifications are merged into the closest classification to avoid the problem of partial matrix data scarcity during subsequent matrix division.

Then, the attribute interaction relationship is obtained according to the attribute classification, and the matrix is divided. For example, if there are 8 user attribute categories, male and female, $<$ 24 years old, 24–34 years old, $>$ 34 years old, low income, middle income, high income, there are 3 item attribute categories: romantic, action, disaster, Then we can obtain 24 attribute interaction pairs, namely (male, romantic), (female, romantic), etc., each attribute pair is associated with a sub-matrix. For each rating $r_{ui}$ in the original matrix, it can be divided into different sub-matrices according to the attributes of user $u$ and item $i$ .

3.3 Sub-matrices training and optimization

There are two important hyperparameters when training a basic MF model, regularization coefficients and hidden vector dimensions. In AIMF, multiple sub-matrices are generated after the original matrix is divided, and the sizes of the sub-matrices are different, so the settings of these two hyperparameters also need to be adjusted adaptively. Two optimization strategies for sub-matrix training are proposed below.

3.3.1 Optimization strategy of regularization coefficient

When a machine learning model is trained, regularization coefficients are used to limit the feature space, and its size is negatively correlated with the number of training data sets. A proper regularization coefficient can theoretically make the model get better results in training. In AIMF, the size of each sub-matrix is different, and sometimes the difference is particularly large. Assuming that the regularization coefficient used in the original MF is $\lambda$ , since the amount of data contained in each sub-matrix must be less than or equal to that of the original rating matrix, the regularization coefficient of each sub-matrix is preferably greater than the regularization coefficient of the original matrix. The smaller the amount of sub-matrix data, the larger the corresponding regularization coefficient needs to be set. A strategy is proposed to set the regularization coefficients of different sub-matrices.

$\displaystyle\lambda^{j}=\lambda\left(1+lg\frac{\sum_{u=1}^{m^{j}}\sum_{i=1}^{% n^{j}}I(r_{ui}^{j})}{\sum_{u=1}^{m}\sum_{i=1}^{n}I(r_{ui})}\right)$ (4)

where $\lambda^{j}$ represents the regularization coefficient of the $j^{\text{th}}$ sub-matrix factorization, $m^{j}$ and $n^{j}$ represent the number of users and items in the sub-matrix, $I$ is an indicator function, it return 0 when the rating value is null, otherwise, it returns 1. $\sum_{u=1}^{m}\sum_{i=1}^{n}I(r_{ui})$ and $\sum_{u=1}^{m^{j}}\sum_{i=1}^{n^{j}}I(r_{ui}^{j})$ refer to the number of effective ratings in the original rating matrix and the amount of effective ratings in the $j^{\text{th}}$ sub-matrix, respectively. Generally, the size of the sub-matrix is normally distributed. There are some sub-matrices, and their sizes are very different from the original matrix. To reduce the impact of the huge difference in the number of matrices, $l g$ is used here to shrink the influence. The main purpose is to make the change of $\lambda$ within an acceptable range. The matrix factorization iteration formula after adopting the optimization strategy of regularization coefficient is defined as

$\displaystyle q_{i}^{j}=q_{i}^{j}+\gamma(e_{ui}^{j}p_{u}^{j}-\lambda^{j}q_{i}^% {j})$ (5) $\displaystyle p_{u}^{j}=p_{u}^{j}+\gamma(e_{ui}^{j}q_{i}^{j}-\lambda^{j}p_{u}^% {j})$ (6)

where $e_{ui}^{j}=r_{ui}^{j}-\hat{r}_{ui}^{j}$ , $q_{i}^{j}$ and $p_{u}^{j}$ represent the latent vector of item $i$ and user $u$ in $j^{\text{th}}$ sub-matrix.

[b] Matrix partitioning and sub-matrix training[1] $R$ , $A$ , $A u$ , $A i$ , $z$ , $m$ , $n$ , $k$ , $\gamma$ , $\lambda$ , num_iter //rating matrix, attribute interaction set, user attribute set, item attribute set, the number of attribute interactions, number of users, number of items, the length of latent vector, learning rate, regularization coefficient, number of training iterations subV //latent vector set of users and items initialize subM; //sub-matrix set ( $j=0$ ; $j<z$ ; $j++$ ) ( $s=0$ ; $s<m$ ; $s++$ ) //matrix partitioning ( $t=0$ ; $t<n$ ; $t++$ ) if ( $A[j]$ .equal ( $A_{u}[s]$ , $A_{i}[t]$ )) addToSub ( $\textit{subM}[j]$ , $s$ , $t$ , $R[s][t]$ ); ForkProcess_Run $\textit{subM}[j]$ ; //run as a new process

return subV;

ForkProcess_Run $\textit{subM}[j]$ //sub-matrix training process initialize $\textit{subV}[j]$ , $p[j]$ , $q[j]$ ; update $\lambda[j]$ and $k[j]$ according to the optimization strategies; (int $\textit{iter}=0$ ; $\textit{iter}<\textit{num\_iter}$ ; $\textit{iter}++$ ) ( $r[j][u][i]$ in $\textit{subM}[j]$ ) $e[j][u][i]=r[j][u][i]-p[j][u]*q[j][i]$ ; $q[j][i]\leftarrow q[j][i]+\gamma(e[j][u][i]*p[j][u]-\lambda[j]*q[j][i])$ ; $p[j][u]\leftarrow p[j][u]+\gamma(e[j][u][i]*q[j][i]-\lambda[j]*p[j][u])$ ; $\textit{subV}[j]$ .updateuser ( $u$ , $p[j][u]$ ); $\textit{subV}[j]$ .updateitem ( $i$ , $q[j][i]$ );

3.3.2 Optimization strategy of latent vector dimension

Hidden vectors represent the characteristics of a user or product. The larger the dimension of a hidden vector, the richer the meaning of the features it can represent. At the same time, the larger the training data set, the more the value of the hidden vector obtained can reflect the true characteristics of the user or item. Some users and products in the rating matrix are relatively low in activity, while others are very high. If the same $k$ value is used to train sub-matrices of different sizes, it is difficult to effectively reflect the characteristics of users and products with high activity, and users and products with low activity will lack hidden data to train hidden vectors, resulting in their Implicit vector underfitting.

The size of each sub-matrix in the AIMF model is different. In this paper, a $k$ value that is suitable for the magnitude of the sub-matrix data is used for matrix decomposition training. Under this method, the hidden vectors obtained by training are relatively more efficient for training data. Therefore, before training, the adaptive $k$ value must be obtained first. The adaptive hidden vector dimension of the $j^{\text{th}}$ sub-matrix can be expressed as:

$\displaystyle k^{j}=\left\lceil k\frac{lg\sum_{u=1}^{m^{j}}\sum_{i=1}^{n^{j}}I% (r_{ui}^{j})}{lg\sum_{u=1}^{m}\sum_{i=1}^{n}I(r_{ui})}\right\rceil$ (7)

where $k$ is the hidden vector dimension of the original rating matrix, which is a hyperparameter and needs to be adjusted according to the actual experimental results in the experiment.

To improve model training efficiency, the usual way is to use parallel processing technology. The key to the traditional parallel matrix decomposition training is the data synchronization during processing. In AIMF method, the original matrix has been divided into several sub-matrices with smaller data volume. These sub-matrices can be trained in parallel, without interfering with each other during training, without the need to consider data synchronization.

In this paper, a stochastic gradient descent algorithm is used to train the sub-matrix factorization model. The latent vectors obtained by each sub-matrix factorization are stored separately and then combined. The specific process is shown in Algorithm 1.

Algorithm 1 contains 2 parts. The first part is matrix partitioning. After the first part is completed, all sub-matrices will be returned. The time complexity of the first part is $O(|A|\cdot m\cdot n)$ . The second part is running in a multi-process environment. The final training time is determined by the one with the longest training time among these matrices, assuming the $j^{\text{th}}$ sub-matrix. The time complexity of training of sub-matrices is $O(\textit{num\_iter}\cdot\textit{subM}[j].\textit{size}())$ . Generally, the size of the sub-matrices is much smaller than the original matrix, and the value of $|A|$ is also much smaller than the value of num_iter, so the time complexity of Algorithm 1 is the model training time corresponding to the largest sub-matrix.

3.4 Latent vectors merging

For the basic MF model, after the model training is completed, a hidden vector will be generated for each user and each product. By multiplying the hidden vector of user u and product $i$ , we can get the prediction of user $u$ ’s rating for the product $i$ . This predicted value can approximate the original rating matrix and fill in the missing data in the original rating matrix. In AIMF, each user and product will correspond to several hidden vectors, and the number of latent vectors is the number of attribute interactions. For example, if a user has a relationship with 9 attribute interactions, then the user will have 9 hidden vectors. If the three attribute interactions are shared between user $u$ and item $i$ , then when calculating the score of user $u$ on item $i$ , so we can get three ratings.

Latent vector merging refers to effectively combining multiple sets of hidden vectors obtained by sub-matrix training. In practical application, there will be a phenomenon: the size of some sub-matrices is relatively large and some are relatively small. The latent vectors generated by large-scale matrix training have a greater impact on the overall prediction performance, so the proportion of their ratings should be increased when merging.

Here we propose several different weight distribution methods for measuring three different prediction ratings, how to choose the most influential rating, how to strengthen the influential rating, and how to synthesize these ratings for prediction.

Figure 3.

An instance of latent vectors merging.

Figure 3 illustrates the process of latent vectors merging and rating prediction through an example. Among them, user $u$ is related to attribute interaction 1, 2, 3, 4, and 5 while item $i$ is related to attribute interaction 2, 3, 4, 6, and 7, i.e., user $u$ and item $i$ share attribute interaction 2, 3, and 4. Here we put forward several different weight distribution methods to merge these ratings of sub-matrices. Suppose $w_{ui}^{j}$ is the rating related to the $j^{\text{th}}$ sub-matric for user $u$ and item $i$ .

(1)

Dist I: Average distribution method. it is the simplest merging method.

$\displaystyle w_{ui}^{j}=\frac{1}{N_{ui}}$ (8)

where $N_{ui}$ is the number of attribute interactions that user $u$ and item $i$ share. This kind of distribution method does not consider the data distribution of sub-matrices and the relationship between attribute interactions. It is only used as a baseline for comparison.

(2)

Dist II: Distribution method based on user activity. In AIMF, the partitioning of sub-matrices is determined by attribute interactions and different attribute interactions show different levels of importance. If the interaction of a certain attribute is very important, then its corresponding hidden vector should be assigned a higher weight when combined. Therefore, how to measure the importance of attribute interaction is the key. Here, we assume that the higher the proportion of a user who rates an item in a sub-matrix or the more frequent the rating behavior, the more the user is interested in the attribute interaction. Conversely, if a user has few ratings in a sub-matrix, it means that this attribute interaction is not important to the user. In the combination of hidden vectors, the sub-matrix that the user pays more attention to can be set with a higher weight. Based on this idea, the distribution method based on user activity is defined as follows.

$\displaystyle w_{ui}^{j}=\frac{1+\sum_{t=1}^{n^{j}}I(r_{ut}^{j})}{lg(n^{j})}$ (9)

where $n^{j}$ is the item number in the $j^{\text{th}}$ sub-matrix, $\sum_{t=1}^{n^{j}}I(r_{ut}^{j})$ is the number of items which are rated in the $j^{\text{th}}$ sub-matrix, 1 is a smoothing coefficient (can be adjusted according to the actual situation), which is used to ensure that each prediction rating has a certain weight value under special circumstances. Since the number of items in the matrix is very large, we take the logarithm of $n^{j}$ to reduce the influence of the number of products in the matrix on the overall weight.

(3)

Dist III: Distribution method based on item activity. Since how to measure the importance of attribute interaction is the key. Here, we assume that when an item gets a higher rating ratio in a sub-matrix, it indicates that the item is more popular for this attribute interaction, so we can set higher weight for its weight. Based on this idea, the distribution method based on item activity is defined as follows.

$\displaystyle w_{ui}^{j}=\frac{1+\sum_{s=1}^{m^{j}}I(r_{si}^{j})}{lg(m^{j})}$ (10)

where $m^{j}$ is the user number in the $j^{\text{th}}$ sub-matric, $\sum_{s=1}^{m^{j}}I(r_{si}^{j})$ is the number of users who rate on item $i$ in the $j^{\text{th}}$ sub-matrix. This can be regarded as the proportion of the evaluation of the product $i$ in the attribute interaction $j$ matrix to the total number of users, and also reflects the activity of the product $i$ in the attribute interaction. This can be regarded as the proportion of the number of users whose item $i$ is evaluated in the $j^{\text{th}}$ matrix to the total number of users, and also reflects the activity of item $i$ in this attribute interaction.

(4)

Dist IV: Distribution method based on user and item activity. In this method, two factors of user activity and item activity are both considered.

$\displaystyle w_{ui}^{j}=\frac{1+\sum_{t=1}^{n^{j}}I(r_{ut}^{j})}{lg(n^{j})}% \cdot\frac{1+\sum_{s=1}^{m^{j}}I(r_{si}^{j})}{lg(m^{j})}$ (11)

(5)

Dist V: Further adjustment. In addition to the preferences of the user and the item itself, this method also takes into account the credibility of each attribute interaction matrix. For example, the greater the amount of data in an attribute interaction matrix, the higher the reliability of the latent vectors trained under the matrix, and the lower the reliability of the latent vectors trained by the matrix with the smaller amount of data.

$\displaystyle w_{ui}^{j}=\frac{1+\sum_{t=1}^{n^{j}}I(r_{ut}^{j})}{lg(n^{j})}% \cdot\frac{1+\sum_{s=1}^{m^{j}}I(r_{si}^{j})}{lg(m^{j})}\cdot lg\left(1+\sum% \nolimits_{s=1}^{m^{j}}\sum\nolimits_{t=1}^{n^{j}}I(r_{st}^{j})\right)$ (12)

where $\sum_{s=1}^{m^{j}}\sum_{t=1}^{n^{j}}I(r_{st}^{j})$ refers to the rating number in $R^{j}$ . After the distribution weights are determined, the values need to be normalized. In this paper, softmax is used for normalization, which further improves the weight of predictive scoring with high impact. The normalized formula of weights is as follows.

$\displaystyle{w^{\prime}}_{ui}^{j}=\frac{e^{w_{ui}^{j}}}{\sum_{t}^{N_{ui}}e^{w% _{ui}^{t}}}$ (13)

After the weights are normalized, we can use them to calculate the prediction ratings, the formula is as follows.

$\displaystyle\hat{r}_{ui}=\sum_{j=1}^{z_{ui}}{w^{\prime}}_{ui}^{j}(q_{i}^{j})^% {T}p_{u}^{j}$ (14)

4. Experimental results

In this section, we evaluate the performance of the proposed method AIMF on two real-world datasets and examine the effect of different parts of AIMF. We conduct experiments with the aim of answering the following questions.

•
RQ1 How do the different latent vectors merging methods impact AIMF?
•
RQ2 How does the optimization of latent vectors dimension impact AIMF?
•
RQ3 How does the optimization of regularization coefficient impact AIMF?
•
RQ4 How does AIMF perform compared to other MF-based methods?
•
RQ5 How does AIMF reduce the training time?

4.1 Experiment settings

Datasets. Two public datasets are employed in our experiments. Both MovieLens-100K and MovieLens-1M datasets were collected under the GroupLens Research Project at the University of Minnesota. The former contains 100,000 ratings for 1,682 movies by 943 users. It also contains some attribute information such as age, gender, occupation for users and release date, and genres for movies. The latter contains 1,000,209 ratings from 6,040 users for 3,900 movies. It has the same attribute information as MovieLens-100K [20].

Baselines. The methods in our comparative evaluation are the follows.

(1)
MF. The basic MF model is mainly used as a benchmark for comparison.
(2)
SVD++. Based on the basic MF model, this model considers users historical rating records, so that it can learn the users latent interest preferences, and improve the performance without adding more training data.
(3)
UMF. Based on the basic MF model, this model considers users attribute information. Equation (2) shows the rating prediction.
(4)
IMF. Based on the basic MF model, this model considers items attribute information. Equation (3) shows the rating prediction.
(5)
AIMF. It is the method proposed in this paper and no optimization is done on it.
(6)
AIMF-1. It is based on AIMF and the strategy of adaptive regularization coefficient optimization is applied.
(7)
AIMF-2. It is based on AIMF and the strategy of adaptive latent vector dimension optimization is applied.

Regulation. We use L2 regularization.

Hyperparameters. The model-independent hyperparameters are set as follows. The initial learning rate is set to 0.01. Its value decays to 95% every 50 iterations. The initial latent vector dimension $k$ is set to 5 and the initial regularization coefficient is set to 0.005. In AIMF-1 and AIMF-2 with optimization strategies, different $k$ and lambda are calculated for different sub-matrices training. The number of training iterations is set to 300.

Platform and implementation. All experiments are conducted in a server with Ubuntu 16.04. The configuration is as follows: 2 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10 GHz, Nvidia Tesla K80 GPU, and 128 GB memory. We implement all methods in python3.6. Due to the limited size of the dataset, we use the cross-validation method to improve the training performance of the model. In each experiment, the training dataset and the test dataset were randomly selected from the original dataset with a ratio of 8:2. All experiments are repeated 5 times. We take the average of these 5 experimental results as the final result.

Evaluation metrics. We use two conventional evaluation metrics, Root Mean Square Error (RMSE) and recommendation Hit Ratio (HR). The latter is divided into two types, not considering recommendation order ( $HR_{1}$ ) and considering recommendation order ( $HR_{2}$ ).

Root Mean Square Error (RMSE) is defined as

$\displaystyle\textit{RMSE}=\sqrt{\frac{1}{M}\sum_{u=1}^{M}(h(x_{u})-y_{u})^{2}}$ (15)

where $M$ is the number of observations, $y_{u}$ refers to the true value of each observation, and $h(xu)$ refers to the prediction calculated by the model. RMSE is used to evaluate the overall accuracy of the model. The smaller the RMSE value, the higher the overall accuracy of the model.

Recommendation Hit Rate (HR) is divided into two types: not considering the order of recommendation ( $HR_{1}$ ) and considering the order of recommendation ( $HR_{2}$ ).

$HR_{1}$ is defined as

$\displaystyle{HR}_{1}=\frac{1}{M}\sum_{u=1}^{M}\frac{1}{L}\left(\sum_{i=1}^{L}% f_{1}(\textit{Pred}(u,i)\in\textit{Real}(u))\right)$ (16)

where $f_{1}(\cdot)$ is a decision function, when the input is true, it returns 1, otherwise it returns 0. $\textit{Pred}(u,i)$ represents the i-th item in the list of top-K items recommended for user $u$ , and $\textit{Real}(u)$ represents the set of real top-K items user $u$ access. $L$ is the number of recommended items.

$HR_{2}$ is defined as

$\displaystyle{HR}_{2}=\frac{1}{M}\sum_{u=1}^{M}\frac{1}{L}\left(\sum_{i=1}^{L}% f_{2}(\textit{Pred}(u,i)\in\textit{Clus}(u,i))\right)$ (17)

Since users may rate the same value for multiple items, the computation of $HR_{2}$ becomes complicated. If the ratings of the items recommended for user $u$ are different, then the recommendation order of each item is unique, ranked from high to low. Otherwise, there may be multiple recommendation order for some items. $\textit{Clus}(u,i)$ represents a set of orders for item $i$ of user $u$ .
4.2 Performance

4.2.1 Evaluation of merging methods (RQ1)

We compare and analyze the impact of different latent vector merging methods in AIMF based on RMSE, as shown in Table 2. Different merging methods show different performances, however, the trend of performance change is consistent. We rank the methods according to performance from high to low, and they are Dist V $>$ Dist IV $>$ Dist III $>$ Dist II $>$ Dist I. Dist I is the simplest method to implement, it considers the least factors, so it has the worst effect. Dist II and Dist III separately consider the activity of users and items, and the two methods perform slightly differently in different datasets. Dist IV is a combination of Dist II and Dist III, and performs better than both of them. Dist V performs best, so in the next experiment, we use Dist V as the merging method in AIMF and compare it with other MF-based methods.

Table 2
Comparison of different latent vector merging methods

AIMF with different latent vectors merging methods	RMAE
	Movielens 1M	Movielens 100k
AIMF (Dist I)	0.9162	0.9567
AIMF (Dist II)	0.9092	0.9321
AIMF (Dist III)	0.9089	0.9347
AIMF (Dist IV)	0.9057	0.9272
AIMF (Dist V)	0.9023	0.9215

4.2.2 Optimization evaluation of latent vector dimension (RQ2)

To verify the effectiveness of the optimization strategy about latent vector dimension proposed in this paper, multiple sets of different values are set the hyperparameter $K$ in the experiment to adjust the $K$ value of each sub-matrix. Table 3 shows the performance comparison of AIMF using different latent vector dimensions. First, when $K$ is set to 5, the RMSE values of AIMF on the two datasets Movielens-1M and Movielens-100K are decreased by 2.42% and 6.58% respectively compared to the basic MF model, which reflects the advantages of AIMF. The RMSE values of AIMF-2 on the two datasets are decreased by 4.68% and 9.66% respectively compared to the basic MF model, which reflects the advantages of AIMF-2. Next, using different $K$ values, the performance of the AIMF-2 method is also different. We find that when $K=5$ , AIMF-2 performs best. At this time, the latent vector dimension of each sub-matrix is determined by the amount of data, and the maximum value is 5. Therefore, in the following experiments, we set $K=5$ in AIMF, AIMF-1 and AIMF-2.

Table 3
Comparison of different latent vector merging methods

Models with different K values	RMAE
	Movielens 1M	Movielens 100k
MF (K $=$ 5)	0.9247	0.9865
AIMF (K $=$ 5)	0.9023	0.9215
AIMF-2 (K $=$ 3)	0.9432	0.9577
AIMF-2 (K $=$ 4)	0.9054	0.9278
AIMF-2 (K $=$ 5)	0.8843	0.8912
AIMF-2 (K $=$ 6)	0.9091	0.9347
AIMF-2 (K $=$ 7)	0.9362	0.9521

4.2.3 Optimization evaluation of regularization coefficient (RQ3)

To verify the effectiveness of the regularization coefficient optimization strategy proposed in this paper, multiple sets of different regularization coefficients are set in this paper. By adjusting $\lambda$ , the regularization coefficients of each sub-matrix can be adjusted. It can be found from Table 4 that under the same setting of $\lambda$ as 0.005, the effect of AIMF-1 has been improved compared to AIMF, and the RMSE values on Movielens-1M and Movielens-100K data sets have decreased by 0.91% and 1.11%. On Movielens-1M dataset, AIMF-1 performed best when $\lambda$ is 0.004, and its RMSE decreased by 4.13% compared to MF. On dataset Movielens-100K, AIMF-1 performed best when $\lambda$ is 0.003, and its RMSE value decreased by 9.45% compared to MF. So in the next experiments, we set $\lambda$ of AIMF-1 to 0.004 in dataset Movielens-1M and 0.003 in dataset Movielens-100K.

Table 4
Comparison of AIMF using different regularization coefficients

Models with different $\lambda$ values	RMAE
	Movielens 1M	Movielens 100k
MF ( $\lambda=$ 0.005)	0.9247	0.9865
AIMF ( $\lambda=$ 0.005)	0.9023	0.9215
AIMF-1 ( $\lambda=$ 0.005)	0.8941	0.9113
AIMF-1 ( $\lambda=$ 0.004)	0.8865	0.9036
AIMF-1 ( $\lambda=$ 0.003)	0.8977	0.8932
AIMF-1 ( $\lambda=$ 0.002)	0.9098	0.9071
AIMF-1 ( $\lambda=$ 0.001)	0.9142	0.9182
AIMF-1 ( $\lambda=$ 0.006)	0.8992	0.9136
AIMF-1 ( $\lambda=$ 0.007)	0.9081	0.9195
AIMF-1 ( $\lambda=$ 0.008)	0.9162	0.9265

4.2.4 Evaluation of method performance (RQ4)

As can be seen from Table 5, the performance of MF is the worst, and the performance of SVD++ has been improved because users implicit feedback information is combined in the model, and UMF and IMF obtain similar performance to SVD++ users attribute information and items attribute information are combined in the models respectively. From a subtle point of view, the effect of UMF is slightly better than IMF, because the users attribute information in the data set is more abundant than the items attribute information in these two datasets. The improvement of AIMF is higher than that of SVD++, UMF and IMF. The effects of AIMF-1 optimized by regularization coefficients and AIMF-2 optimized by hidden vector dimensions have been significantly improved. Among them, AIMF-2 performed best. Compared with AIMF-1, RMSE values of AIMF-2 on the two datasets were reduced by 0.24% and 0.22%, respective.

Table 5
RMSE comparison of MF-based methods

Models	RMAE
	Movielens 1M	Movielens 100k
MF	0.9247	0.9865
SVD++	0.9132	0.9473
UMF	0.9146	0.9467
IMF	0.9167	0.9517
AIMF	0.9023	0.9215
AIMF-1	0.8865	0.8932
AIMF-2	0.8843	0.8912

The metric of RMSE is mainly used to evaluate the overall effect of machine learning models while the metric of HR is more suitable for the actual needs of the recommender systems. Next, we will show the results of $HR_{1}$ and $HR_{2}$ . Through comparative analysis, we can judge which MF-based method performs better in recommendation task.

Table 6

Comparison of MF-based methods for hit ratio not considering recommendation order

Models	Movielens-1M				Movielens-100k
	1	5	10	15	1	5	10	15
MF	0.1064	0.2723	0.3723	0.4709	0	0.2041	0.3202	0.4152
SVD++	0.1064	0.3015	0.3832	0.4766	0	0.2123	0.3221	0.4228
UMF	0.0638	0.3011	0.3854	0.4771	0	0.2166	0.3238	0.4236
IMF	0.1064	0.2936	0.3821	0.4708	0.02	0.1916	0.3212	0.4236
AIMF	0.1277	0.3191	0.3915	0.4894	0	0.225	0.3542	0.4444
AIMF-1	0.1523	0.3439	0.4295	0.5020	0	0.2595	0.3822	0.4615
AIMF-2	0.1538	0.3423	0.4317	0.5026	0	0.2631	0.3889	0.4692

It can be seen from Table 6 that as the length of the recommendation list increases, the value of $HR_{1}$ will also increase. As a whole, the order of $HR_{1}$ from high to low among the various methods is: AIMF-2 $>$ AIMF-1 $>$ AIMF $>$ UMF $>$ IMF $>$ SVD++ $>$ MF. The experimental results are consistent with RMSE. Table 7 presents $HR_{2}$ of different MF-based methods on the two datasets. As a whole, the order of $HR_{2}$ from high to low among the various methods is also: AIMF-2 $>$ AIMF-1 $>$ AIMF $>$ UMF $>$ IMF $>$ SVD++ $>$ MF. Both metrics are consistent.

By comparing Tables 6 and 7, we can find that $HR_{2}$ is smaller than $HR_{1}$ under the same environment. When the recommendation list is shorter, the recommendation results are more unstable and will be affected by various random factors. When the recommendation list is longer, the recommendation results are more stable and the performance is better. It can be found from Table 6 that when the length of the recommendation list is 15, compared to the basic MF model, the $HR_{2}$ values of AIMF-1 on two datasets are improved 7.64%, 7.79%, respectively, $HR_{2}$ values on AIMF-2 are improved 11.91%, and 12.87%, respectively.

4.2.5 Training time of models (RQ5)

In addition to the accuracy of the evaluation method, we also make statistics on the training time of each method. The specific values are shown in Table 8. From Table 8, we can see that compared to MF, the time spent in SVD++, UMF, and IMF training generally increases, because these methods require training additional hidden vector parameters. Although the AIMF method requires more training parameters than MF, AIMF can be performed in a parallel environment. In the experiment, each sub-matrix is assigned a parallel task, so the overall training time is determined by the sub-matrix with the longest training time. When the computing resources are sufficient and the amount of data between the sub-matrices is not much different, the parallel processing technique in this paper can help shorten the training time effectively. Compared with MF, the time consumption of AIMF on the two data sets is reduced by 65.11% and 53.44% respectively.

Table 7
Comparison of MF-based methods for hit ratio considering recommendation order

Models	Movielens-1M				Movielens-100k
	1	5	10	15	1	5	10	15
MF	0.1064	0.2553	0.3021	0.3361	0	0.1917	0.2937	0.3333
SVD++	0.1064	0.2726	0.3024	0.3359	0	0.2016	0.2866	0.3272
UMF	0.0638	0.2779	0.3059	0.3396	0	0.2041	0.2954	0.3319
IMF	0.1064	0.2718	0.3098	0.3354	0.02	0.1997	0.2855	0.3305
AIMF	0.1277	0.2894	0.3104	0.3448	0	0.2167	0.3146	0.3569
AIMF-1	0.1523	0.3120	0.3365	0.3618	0	0.2389	0.3350	0.3732
AIMF-2	0.1538	0.3221	0.3366	0.3623	0	0.2416	0.3366	0.3762

Table 8

Training time comparison of different MF-based methods

Models	Running time (s)
	Movielens 1M	Movielens 100k
MF	731	189
SVD++	882	267
UMF	856	243
IMF	812	221
AIMF	255	88
AIMF-1	257	89
AIMF-2	256	88

5. Conclusion and future directions

To exploit the advantages of the user and item attributes for matrix factorization in rating prediction and recommendation tasks, in this paper we propose an attribute interaction-aware matrix factorization method AIMF. In this method, the original matrix is divided into multiple sub-matrices that are associated with particular attribute interactions, and these sub-matrices are separately trained and then the latent vectors generated by them are fused. This method can not only help discover the effect of attribute interaction on matrix decomposition to improve the performance of matrix factorization but also can use parallel processing to achieve efficient training. Experiments have verified that AIMF method obtains a good performance improvement by directly using the basic MF model. The existing MF optimization algorithms can also be applied to this method to further improve performance. Whether the existing optimization algorithm combined with this method can play a better effect, and how much effect can be improved need further study.

References

Smith

and Linden

, Two decades of recommender systems at amazon.com, IEEE Internet Computing 21(3) (2016), 12–18.

Gomez-Uribe

C.A.

and Hunt

, The netflix recommender system: algorithms, business value, and innovation, ACM Transactions on Management Information Systems 6(4) (2015), 1–19.

Quadrana

Cremonesi

and Jannach

, Sequence-aware recommender systems, ACM Computing Surveys 51(4) (2018), 1–36.

Dara

Chowdary

C.R.

and Kumar

, A survey on group recommender systems, Journal of Intelligent Information Systems 54(2) (2020), 271–295.

Zhang

Chow

and Xu

, Enabling kernel-based attribute-aware matrix factorization for rating prediction, IEEE Transactions on Knowledge and Data Engineering 29(4) (2016), 798–812.

Yuan

Han

and Qian

, Singular value decomposition-based recommendation using imputed data, Knowledge-Based Systems 163 (2019), 485–494.

Guo

Yin

Zhang

and He

, Cold start recommendation based on attribute-fused singular value decomposition, IEEE Access 7 (2019), 11349–11359.

Wang

and Gao

, Attributes coupling based matrix factorization for item recommendation, Applied Intelligence 46(3) (2017), 521–533.

Rendle

, Factorization machines, in: 2010 IEEE International Conference on Data Mining 2010, pp. 995–1000.

10.

Juan

Zhang

Chin

and Lin

, Field-aware factorization machines for CTR prediction, in: Proceedings of the 10th ACM Conference on Recommender Systems, 2016, pp. 43–50.

11.

Yan

Zhang

Zhao

and Huang

, An intelligent field-aware factorization machine model, in: Proceedings of 22nd International Conference on Database Systems for Advanced Applications, 2017, pp. 309–323.

12.

Zhang

Liu

Song

and Sun

, Nonlinear non-negative matrix factorization using deep learning, in: Proceedings of the 2016 International Joint Conference on Neural Networks, 2016, pp. 477–482.

13.

Mnih A.

S.R.R.

,Probabilistic matrix factorization, in: in: Proceedings of the 21th Annual Conference on Neural Information Processing Systems,2008, pp. 1257–1264.

14.

Sharifi

Rezghi

and Nasiri

, New algorithm for recommender systems based on singular value decomposition method, in: Proceedings of the 3rd International Conference on Computer and Knowledge Engineering, 2013, pp. 86–91.

15.

Koren

Bell

and Volinsky

, Matrix factorization techniques for recommender systems, Computer 42(8) (2009), 30–37.

16.

Zhou

Liu

Lyu

M.R.

and King

, Recommender systems with social regularization, in: Proceedings of the Fourth International Conference on Web Search and Data Mining, 2011, pp. 287–296.

17.

Zhang

Liu

Zhang

and Shen

, Fused matrix factorization with multi-tag, social and geographical influences for POI recommendation, World Wide Web 22(3) (2019), 1135–1150.

18.

Chen

H.H.

and Chen

, Differentiating regularization weights – a simple mechanism to alleviate cold start in recommender systems, ACM Transactions on Knowledge Discovery from Data 13(1) (2019), 1–22.

19.

Yan

and Zhang

, Merging visual features and temporal dynamics in sequential recommendation, Neurocomputing 362 (2019), 11–18.

20.

K.J.A. Harper F.M., The movielens datasets: History and context, ACM Transactions on Interactive Intelligent Systems 5(4) (2016), 1–19.

Attribute interaction aware matrix factorization method for recommendation

Abstract

Keywords

1. Introduction

3.1 Framework

Table 1 Notation information

3.3 Sub-matrices training and optimization

3.3.1 Optimization strategy of regularization coefficient

4.2.1 Evaluation of merging methods (RQ1)

Table 2 Comparison of different latent vector merging methods

Table 3 Comparison of different latent vector merging methods

Table 4 Comparison of AIMF using different regularization coefficients

Table 5 RMSE comparison of MF-based methods

Table 7 Comparison of MF-based methods for hit ratio considering recommendation order

References

Table 1
Notation information

Table 2
Comparison of different latent vector merging methods

Table 3
Comparison of different latent vector merging methods

Table 4
Comparison of AIMF using different regularization coefficients

Table 5
RMSE comparison of MF-based methods

Table 7
Comparison of MF-based methods for hit ratio considering recommendation order