Skip to content

Make the evaluation metric configurable  #13

@bkersbergen

Description

@bkersbergen

The evaluation metric (MRR@20) that is used for hyperparameter search is currently hardcoded.
Serenade has supports for multiple evaluation metrics such as MRR, F1, HitRate, Ndcg, Popularity. Precision, Recall etc.

The choice for which evaluation metric to be used, should be configurable by the user via the configuration file.

Proposed solution:
in the config file introduce a new variable for the section 'hyperparam' used by tpe_hyperparameter_optm.rs:
metric_name = "MRR"
metric_length = 20

in the config file introduce a new section that evaluate_file.rs should use:
[evaluation]
metric_name = "MRR"
metric_length = 20
training_data_path = "train.txt"
test_data_path = "test.txt"

Both the tpe_hyperparameter_optm.rs and evaluate_file.rs should be included to support this feature.

The trait SessionMetric could be extended with a function fn is_larger_better(&self) -> bool
By default all metrics must return true for fn is_larger_better except for popularity.rs which should return false

The tpe_hyperparameter_optm.rs ( line 130 ) should use the 'max' or 'min' score depending on the is_larger_better value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions