Request for paper reproducibility parameters

[maniospas]

Dear authors,
I am a phd student trying to replicate your methodology and would like to clarify your choice of hyperparameters, given that I cannot find an explicit description and seem to obtain different results. Did you use the ones in this repository's readme example? (i.e. --alg_type ngcf --regs [1e-5] --lr 0.0001 --pretrain 0 --batch_size 4096 --epoch 800 and with default parameters for the rest of the code)

For the sake of a sanity check, could you also provide evaluation outcomes for one of the dataset splits that are uploaded in this repository? (e.g for qiaoji_1_1 given that qiaoji_5_30 has not been uploaded)

Best regards and thank you in advance for your time,
Manios

[Limber0117]

Hi Manios,

Thanks for your inquiry and sorry for the confusion caused.

Yes, you are right, most of those hyperparameters not mentioned in our manuscript are using the default values listed in the source code. Indeed, we will need to specify the following parameters during the training.

1. dataset folder name: such as qiaoji_5_3, here the 5 means for each mobile app, we removed randomly 5 libraries to conduct the testing set and the remaining libraries are used as the training set, and the 3 means the 3rd group of training and testing datasets.
2. embedding size, which specifies how large the latency spaces are during the embedding. It can be from 32 to 128 in our experiments. Generally, a larger value leads to higher prediction performance but longer training time consumption.
3. layer size, it is an array, like [64,64,64]. This example means each layer of the GNN network has 64 nodes, and this GNN has a total number of 3 layers. Similarly, the larger the better. But in our experiments, 3 layers with 128 nodes for each layer is enough for GRec to obtain a good performance.
4. epoch, indeed, our approach can stop earlier than the given epoch if it has obtained the optimal results. Thus, it doesn't matter if the given epoch is too large. Usually, epoch > 500 is enough.
5. mess_dropout, it is also an array, with the same dimensions as the layer size, for example [0.1,0.1,0.1]. According to many previous researches, by randomly dropping a few data during the training, GRec can get better performance. However, the value cannot be too large, 0.1 is the default value in our experiments.

I just found that I cannot log into the OZSTAR now, all data and results are stored there. What a shame! But don't worry, you will find the recommendations for each mobile app in the folder "output>qiaoji_x_y", where qiaoji_x_y is equal to the dataset name. You will find something like "0,[145, 281, 547, 605, 607, 250, 191, 172, 694, 295]". Here, the first "0" means the first mobile app, then the rest means the recommended libraries. So, you can compare them against the testing set to check if the results reported by GRec are correct. The results reported in our manuscript can be found in the "result.csv" file in folder "qiaoji_x_y".