How to get the benchmark dataset for evaluation ?

[tangshixiang]

I wonder how to get the benchmark dataset for evaluation part?

You have mentioned to use python src/simplefold/evaluation/analyze_folding.py script when evaluating, but how can I get the data_dir [PATH_TO_TARGET_MMCIF]? Take CASP14 for instance, where can I get all the mmcif files needed for evaluation and fasta files needed for inference?

Best wishes,

Shixiang

[yuyangw]

Hi, thanks for your interest in our work. You can find the target list in CASP14 in the official website: https://predictioncenter.org/casp14/targetlist.cgi and download the mmcif file corresponding to each target in PDB: https://www.rcsb.org/.

[Xinping-Liu]

By the way I wonder how to process the corresponding PDB mmcif file as ground truth? For example, I want 7dz2-A , how to extract the information of the A-chain of 7dz2.cif? I wrote a script to extract the chain by myself, but turn out to found that the prediction you provided didn't match the score you claimed in the paper, in TM-score of cameo22 benchmark regarding 100M model.

[Xinping-Liu]

And some targets of casp14 listed in your paper seem to have no pdb reference, such as T1050, T1101 e.t.c.

[yuyangw]

Hi @Xinping-Liu, thanks for your interest in our work. For CAMEO22, we follow the target list in EigenFold: https://github.com/bjing2016/EigenFold/blob/master/splits/cameo2022.csv. For CASP14, you'll need to query the committee for the target structures.

In PDB files, there are missing components (e.g., atoms and residues). In inference we use the complete amino acid sequence as the input and generate the full structure (same for all the baselines). It is likely that generating structures from amino acid sequence of processed PDBs misses important components and leads to incomparable results.

[Xinping-Liu]

Yeah, thank you for your kind reply.

I checked the list of cameo22, and find that there is a little difference between https://github.com/bjing2016/EigenFold/blob/master/splits/cameo2022.csv and Appendix D.4 TargetsinFoldingTasks in your paper. 8ahp_A is replaced by 8qcw_A, and that makes me confused.

[Xinping-Liu]

And I would appreciate it much if you can tell me how you deal with the ground truth cif files. Incomparable results are produced by downloaded prediction instead of the prediction made by downloaded model or trained model, so I guess it's not the problem of aa sequence or fasta, but the problem of structure files or mmcif.

[lujiarui]

Hi @Xinping-Liu, for the target ID, the PDB entries 8ahp has become obsolete and is replaced by 8qcw.

[yuyangw]

Hi @Xinping-Liu, the structures can be found in the EigenFold repo as well: https://github.com/bjing2016/EigenFold/tree/master/structures.

[Xinping-Liu]

Thank you for your kind reply. My problem is solved.