Inquiry for validation dataset

[qxdu]

I have successfully integrated the basenji2.pth model you supplied. To ensure the accuracy of the provided checkpoints,
I would be grateful if you could provide a concise and accessible test dataset that can be utilized for validation purposes. I have searched for suitable datasets within the original repositories, but the examples provided (e.g., https://github.com/calico/basenji/blob/9e1c2e2f5b1b37ad11cfd2a1486d786d356d78a5/tutorials/train_test.ipynb), appear to be based on a simplified model and do not include the full set of 4229 genomic tracks. Your assistance in providing a validation dataset would be immensely beneficial.

Thank you in advance for your consideration and support.

[d-laub]

Hi Qixiu,

Thanks for looking into this. The only full target data I'm aware of come from the basenji-barnyard GCP bucket which are very large (300+ GB) and the bucket is requestor pays. A cheaper alternative could be computing the correlation between predictions from the tensorflow and pytorch implementations. I think the BED file may be in the original basenji repository, otherwise I'll check if I have it already and can upload it.

[qxdu]

Hi Qixiu,

Thanks for looking into this. The only full target data I'm aware of come from the basenji-barnyard GCP bucket which are very large (300+ GB) and the bucket is requestor pays. A cheaper alternative could be computing the correlation between predictions from the tensorflow and pytorch implementations. I think the BED file may be in the original basenji repository, otherwise I'll check if I have it already and can upload it.

Thank you for your warm generosity. I’ve observed that the Enformer and Basenji2 models yield identical output dimensions and and consistent dimensional information. To further investigate, I explored the enformer-pytorch repository and discovered a convenient dataset for validation. This dataset not only includes genomic sequences but also the associated target genomic tracks. After conducting a comparison between the enformer-pytorch and basenji2-pytorch implementations, here are the findings:

Correlation between Basenji2 and Enformer:  0.8949757814407349
Correlation between Basenji2 and target:  0.6140885949134827
Correlation between Enformer and target:  0.59630286693573

Thus, I believe this could serve as a straightforward validation method, further confirming the effectiveness of the basenji2-pytorch you’ve provided.

In addition, I am very interested in your provided tf_to_torch.py. Since transforming the tensorflow checkpoints to pytorch is an increasingly important scientific issue, did you develop this conversion independently, or did you refer to existing online resources? Could you please be so kind as to provide some referenceable guidance?

Wishing you a merry Christmas.

[d-laub]

That looks promising with the data from the enformer-pytorch repo. I agree it's a good way to validate the re-implementation, and it may be ideal to see metrics on the full test split across all tracks. Would you happen to know what split, genomic coordinates, and track that data corresponds to?

The tf_to_torch.py in this repo is adapted from the one in the enformer-pytorch repo. I believe I only refactored it for clarity. I also remember looking through the keras/tensorflow docs/implementation and testing with some toy models to confirm the array transpositions used as well.

And likewise, happy holidays!

[qxdu]

I doubt that the description from deepmind accurately corresponds to the genomic tracks of data, but enformer-pytorch has not provided descriptions of the data splits and genomic coordinates.

This might be designed by Phil Wang team, as I haven’t seen this validation dataset elsewhere, and you could propose an issue under the enformer-pytorch project.