user and item biases in WRMF and explicit feedback

[dselivanov]

As of 35247b4

without biases

library(rsparse)
library(lgr)
lg = get_logger('rsparse')
lg$set_threshold('debug')
data('movielens100k')
options("rsparse_omp_threads" = 1)

train = movielens100k

set.seed(1)
model = WRMF$new(rank = 10,  lambda = 1, feedback  = 'explicit', solver = 'cholesky', with_bias = FALSE)
user_emb = model$fit_transform(train, n_iter = 10, convergence_tol = -1)

INFO [23:09:40.158] starting factorization with 1 threads
INFO [23:09:40.268] iter 1 loss = 4.4257
INFO [23:09:40.302] iter 2 loss = 1.2200
INFO [23:09:40.332] iter 3 loss = 0.8617
INFO [23:09:40.361] iter 4 loss = 0.7752
INFO [23:09:40.391] iter 5 loss = 0.7398
INFO [23:09:40.420] iter 6 loss = 0.7191
INFO [23:09:40.456] iter 7 loss = 0.7046
INFO [23:09:40.488] iter 8 loss = 0.6935
INFO [23:09:40.522] iter 9 loss = 0.6845
INFO [23:09:40.555] iter 10 loss = 0.6769

with biases

set.seed(1)
model = WRMF$new(rank = 10,  lambda = 1, feedback  = 'explicit', solver = 'cholesky', with_bias = TRUE)
user_emb = model$fit_transform(train, n_iter = 10, convergence_tol = -1)

INFO [23:10:06.605] starting factorization with 1 threads
INFO [23:10:06.637] iter 1 loss = 0.8411
INFO [23:10:06.671] iter 2 loss = 0.6251
INFO [23:10:06.704] iter 3 loss = 0.5950
INFO [23:10:06.736] iter 4 loss = 0.5820
INFO [23:10:06.769] iter 5 loss = 0.5751
INFO [23:10:06.805] iter 6 loss = 0.5712
INFO [23:10:06.840] iter 7 loss = 0.5688
INFO [23:10:06.875] iter 8 loss = 0.5673
INFO [23:10:06.916] iter 9 loss = 0.5663
INFO [23:10:06.951] iter 10 loss = 0.5657

cc @david-cortes

[david-cortes]

With biases plus mean centering:

DEBUG [22:10:21.514] initializing biases 
INFO  [22:10:21.548] starting factorization with 1 threads 
INFO  [22:10:21.582] iter 1 loss = 0.8305 
INFO  [22:10:21.607] iter 2 loss = 0.6170 
INFO  [22:10:21.631] iter 3 loss = 0.5822 
INFO  [22:10:21.655] iter 4 loss = 0.5662 
INFO  [22:10:21.680] iter 5 loss = 0.5568 
INFO  [22:10:21.706] iter 6 loss = 0.5507 
INFO  [22:10:21.730] iter 7 loss = 0.5464 
INFO  [22:10:21.755] iter 8 loss = 0.5431 
INFO  [22:10:21.781] iter 9 loss = 0.5405 
INFO  [22:10:21.806] iter 10 loss = 0.5383

With rnorm init + biases + mean centering:

DEBUG [22:12:08.799] initializing biases 
INFO  [22:12:08.843] starting factorization with 1 threads 
INFO  [22:12:08.883] iter 1 loss = 0.7696 
INFO  [22:12:08.912] iter 2 loss = 0.6211 
INFO  [22:12:08.938] iter 3 loss = 0.5836 
INFO  [22:12:08.963] iter 4 loss = 0.5660 
INFO  [22:12:08.989] iter 5 loss = 0.5557 
INFO  [22:12:09.015] iter 6 loss = 0.5489 
INFO  [22:12:09.041] iter 7 loss = 0.5442 
INFO  [22:12:09.067] iter 8 loss = 0.5408 
INFO  [22:12:09.093] iter 9 loss = 0.5382 
INFO  [22:12:09.120] iter 10 loss = 0.5362

Although, the loss function is taking the regularization incorrectly so these aren't final numbers yet.

[dselivanov]

@david-cortes

Although, the loss function is taking the regularization incorrectly so these aren't final numbers yet.

Could you please elaborate more on that?

[david-cortes]

It had a bug in which it was taking the squared sum of X twice instead of taking Y. Also I think it's adding a row of all-ones into the calculation.

[dselivanov]

It had a bug in which it was taking the squared sum of X twice instead of taking Y

Ok, this fixed now

Also I think it's adding a row of all-ones into the calculation.

Doesn't seem so as arma::span(1, X.n_rows - 1) skips first and last row (ones and biases)
Ah, you are right, need to be -2

[dselivanov]

# no bias
INFO  [14:55:08.288] starting factorization with 1 threads 
INFO  [14:55:08.394] iter 1 loss = 6.0649 
INFO  [14:55:08.424] iter 2 loss = 0.8184 
INFO  [14:55:08.450] iter 3 loss = 0.7426 
INFO  [14:55:08.480] iter 4 loss = 0.7154 
INFO  [14:55:08.511] iter 5 loss = 0.6984 
INFO  [14:55:08.546] iter 6 loss = 0.6861 
INFO  [14:55:08.581] iter 7 loss = 0.6767 
INFO  [14:55:08.618] iter 8 loss = 0.6691 
INFO  [14:55:08.650] iter 9 loss = 0.6629 
INFO  [14:55:08.691] iter 10 loss = 0.6577 

# user + item bias
INFO  [14:55:18.805] starting factorization with 1 threads 
INFO  [14:55:18.838] iter 1 loss = 0.7335 
INFO  [14:55:18.873] iter 2 loss = 0.5918 
INFO  [14:55:18.907] iter 3 loss = 0.5624 
INFO  [14:55:18.943] iter 4 loss = 0.5496 
INFO  [14:55:18.982] iter 5 loss = 0.5427 
INFO  [14:55:19.022] iter 6 loss = 0.5384 
INFO  [14:55:19.064] iter 7 loss = 0.5355 
INFO  [14:55:19.101] iter 8 loss = 0.5338 
INFO  [14:55:19.148] iter 9 loss = 0.5328 
INFO  [14:55:19.184] iter 10 loss = 0.5323 

# user + item bias + better init
DEBUG [15:21:49.763] initializing biases 
INFO  [15:21:49.767] starting factorization with 1 threads 
INFO  [15:21:49.804] iter 1 loss = 0.7281 
INFO  [15:21:49.842] iter 2 loss = 0.5933 
INFO  [15:21:49.880] iter 3 loss = 0.5619 
INFO  [15:21:49.920] iter 4 loss = 0.5484 
INFO  [15:21:49.961] iter 5 loss = 0.5413 
INFO  [15:21:50.000] iter 6 loss = 0.5370 
INFO  [15:21:50.034] iter 7 loss = 0.5341 
INFO  [15:21:50.074] iter 8 loss = 0.5321 
INFO  [15:21:50.116] iter 9 loss = 0.5308 
INFO  [15:21:50.148] iter 10 loss = 0.5298


# user + item bias + better init + global
DEBUG [15:00:05.874] initializing biases 
INFO  [15:00:05.962] starting factorization with 1 threads 
INFO  [15:26:17.413] iter 1 loss = 0.7213 
INFO  [15:26:17.429] iter 2 loss = 0.5798 
INFO  [15:26:17.440] iter 3 loss = 0.5461 
INFO  [15:26:17.451] iter 4 loss = 0.5317 
INFO  [15:26:17.464] iter 5 loss = 0.5241 
INFO  [15:26:17.471] iter 6 loss = 0.5194 
INFO  [15:26:17.481] iter 7 loss = 0.5164 
INFO  [15:26:17.493] iter 8 loss = 0.5144 
INFO  [15:26:17.500] iter 9 loss = 0.5130 
INFO  [15:26:17.507] iter 10 loss = 0.5121

[david-cortes]

I think the more correct way of adding the regularization with biases to the loss would be to exclude only the row that has ones while still adding the row that has biases, as the regularization is also applied to the user/item biases.

[david-cortes]

I'm pretty sure there is still some unintended data copying going on with the current approach. I tried timing it with the ML10M data, got these results:

• With biases:

Unit: seconds
                                                                                                                                                                                                                                                             expr
 {     m = rsparse::WRMF$new(rank = 40, lambda = 0.05, dynamic_lambda = TRUE,          feedback = "explicit", with_global_bias = TRUE, with_user_item_bias = TRUE,          solver = "conjugate_gradient")     A = m$fit_transform(ML10M, convergence_tol = -1) }
      min       lq     mean   median       uq      max neval
 16.36217 16.36217 16.36217 16.36217 16.36217 16.36217     1

• Without biases:

Unit: seconds
                                                                                                                                                                                                                                                              expr
 {     m = rsparse::WRMF$new(rank = 40, lambda = 0.05, dynamic_lambda = TRUE,          feedback = "explicit", with_global_bias = TRUE, with_user_item_bias = FALSE,          solver = "conjugate_gradient")     A = m$fit_transform(ML10M, convergence_tol = -1) }
      min       lq     mean   median       uq      max neval
 10.97332 10.97332 10.97332 10.97332 10.97332 10.97332     1

Adding the biases made it take 49% longer to fit the model.

For comparison purposes, these are the same times using the package cmfrec, which has a less efficient approach for the biases (uses rank+1 matrices and copies/replaces bias/constant component at each iteration):

• With biases:

Unit: seconds
                                                                                                                                                   expr
 {     m = CMF(X = ML10M, k = 40, lambda = 0.05, scale_lam = TRUE,          use_cg = TRUE, finalize_chol = FALSE, precompute_for_predictions = FALSE) }
     min      lq    mean  median      uq     max neval
 8.07948 8.07948 8.07948 8.07948 8.07948 8.07948     1

• Without biases:

Unit: seconds
                                                                                                                                                                                                  expr
 {     m = CMF(X = ML10M, k = 40, lambda = 0.05, scale_lam = TRUE,          use_cg = TRUE, finalize_chol = FALSE, precompute_for_predictions = FALSE,          user_bias = FALSE, item_bias = FALSE) }
      min       lq     mean   median       uq      max neval
 6.385816 6.385816 6.385816 6.385816 6.385816 6.385816     1

It took only 26% longer with the biases added.

[dselivanov]

@david-cortes should be fixed now, can you try? I'm surprised cmfrec is almost 2x faster! I would expect arma to be translated into effecient blas calls.

[david-cortes]

Now it's producing a segmentation fault.

[david-cortes]

Actually, the segmentation fault was not from 3731ca0, but from the commits that follow. However, this commit actually makes it slower:

Unit: seconds
                                                                                                                                                                                                                                                             expr
 {     m = rsparse::WRMF$new(rank = 40, lambda = 0.05, dynamic_lambda = TRUE,          feedback = "explicit", with_global_bias = TRUE, with_user_item_bias = TRUE,          solver = "conjugate_gradient")     A = m$fit_transform(ML10M, convergence_tol = -1) }
      min       lq     mean   median       uq      max neval
 21.44936 21.44936 21.44936 21.44936 21.44936 21.44936     1

[david-cortes]

Ran it again after using "clean and rebuild" from the current commit at the master branch, now it didn't segfault anymore, and took the same time as before with the biases (~16s). Perhaps I had some issue with the package calling the wrong shared object functions.

[david-cortes]

Another interesting thing however: using float precision makes it take less than half the time when not using biases (~4.7s vs ~10.7s). But when using biases now it returns NANs with float.