Low F1 scores when using default config_finetuning.yaml

[RustT883]

I'm currently doing fine-tuning for RE based on drugprot dataset with 12 possible relations. (https://zenodo.org/records/5042151)

Which config should I use for performing the finetuning procedure? The default config_finetuning.yaml gives low F1 scores (< 0.3) even after increasing num_steps to 20000 and lowering the learning rates. Should I try other configs with fixed_relation_types: true?

Thank you in advance!

[jackboyla]

Hey @RustT883 sorry for the delay! The config_finetuning.yaml may need to be updated; other configs like configs/config_wiki_zsl.yaml are more recent. Let me explain:

• Set fixed_relation_types: true -- this will force the model to consider all 12 possible relation labels for every evaluation instance. If this is false, it only use the labels in a given instance as the possible candidates, which may be too easy if there's only a couple of relations in an instance.
• Take a close look at the data if you haven't already -- specifically, watch out for relations where one direction is annotated (GENE-Y -> ACTIVATOR -> GENE_N) but the reverse direction is not. If the reverse direction shouldn't have a label, then it's fine to not be included, but this can cause train signal noise if it should be there.
• Try to train and evaluate on the same small dev set of ~30 examples, and assert that the model achieves 100% F1. If not, that's a red flag that something in the data is wrong.
• Make sure the entities positions are correct -- token indices should be given, and the end index is inclusive ([see here for an example]https://github.com/jackboyla/GLiREL/tree/main?tab=readme-ov-file#example-training-data)))
• config_finetuning.yaml does not specify a prev_path parameters. This is a checkpoint to continue training from, but since it's not set you're training from the microsoft/deberta-v3-small checkpoint. You may see better performance if you use the model we shared on Huggingface: prev_path: "jackboyla/glirel-large-v0", as this has been already trained on a fairly general dataset.
• I recommend using a larger model to get higher performance, e.g microsoft/deberta-v3-large
• Your dataset does not have a lot of examples : 3500 from what I can see. Due to the specific nature of your dataset's labels, it may be worth generating synthetic data using an LLM, if you have more unlabelled data you wish to use. But only do this once you have done the above sanity checks!

Hopefully this helps - let me know how you get on!