Last edit date: week of Feb 12th, 2024. Note: technical details not mentioned here can be found in TechDocu.md #TODO add [link]
"The most common evaluation task for KGE methods is entity ranking, which is a form of question answering. The available data is partitioned into a set of training, validation, and test triples. Given a test triple (i, k, j) (unseen during training), the entity ranking task is to determine the correct answer—i.e., the missing entity j and i, resp.—to questions (i, k, ?) and (?, k, j).
To do so, potential answer triples are first ranked by their score in descending order. All triples but (i, k, j) that occur in the training, validation, or test data are subsequently filtered out so that other triples known to be true do not affect the ranking. Finally, metrics such as the mean reciprocal rank (MRR) of the true answer or the average HITS@k are computed"
From Ruffinelli et. al, 'You Can Teach an Old Dog new Tricks! On Training Knowledge Grpah Embeddings', ICLR 2020.
We follow this evaluation method to a t, computing both MRR and Hits@k after filtering out other valid answers from the generated output.
#TODO: mention how to run multiple runs with hydra