Questions about Hyperparameters and Protocols for Bitcoin Datasets

[leshanshui-yang]

Hello,

I am a PhD student closely following your article on “CoEvoGNN” for link prediction using the Co-AU 2K and Bitcoin datasets. I greatly appreciate you sharing the associated code on GitHub (https://github.com/DM2-ND/CoEvoGNN).

As I aim to reproduce the results presented in your article using the GitHub repository, I have some questions regarding the experimental aspects that I would like to discuss with you:

1. [Reproducibility]
   When I executed the example.sh directly, the results I obtained (RMSE 1.158, F1max 0.254 for Co-AU 2K ) were worse than the values reported in the paper. I'm curious to know if there are specific hyperparameters or seed values that need to be configured to ensure the reproducibility of the experimental outcomes as described in the paper?

2. [Experimental Protocol]
   Concerning the experiments conducted on the Bitcoin-alpha and OTC datasets, is there any relevant code available? Some details, particularly regarding metrics, seem to be omitted in the article. Specifically, the "f1" values reported for BC alpha (f1 = 0.461) and BC otc (f1 = 0.448) – do these refer to "f1max" as in the Co-AU 2K dataset, or are they related to the "f1" of edge prediction with negative sampling as used in EvolveGCN?

3. [Initial Embedding]
   To achieve the reported results for the Bitcoin-alpha and OTC datasets, is it necessary to initialize H_0 in a manner similar to the Co-AU 2K dataset? If so, could you provide details on how H_0 is calculated?

I would appreciate it if you could share the relevant scripts, dockerfiles, or Colab notebooks of your experiments. This would greatly assist other researchers in reproducing the experimental findings outlined in your article.

Thank you for your time and assistance.

Best regards,

Leshanshui YANG

[mjiang89]

Thanks for your comment, Leshanshui. I've brought it to the first author of the work: Dr. Daheng Wang (dahenwan@amazon.com; dwang8@nd.edu) I asked him to directly address the comment on GitHub.