Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add validation to movie-lens benchmark #441

Merged

Conversation

lovisek
Copy link
Contributor

@lovisek lovisek commented Apr 23, 2024

The benchmark originally returned an empty set of recommended movies due to passing a userID of value 0 to the predict method. User ID 0 is undefined (as the model was not trained with ratings from this user), causing the predict method to return an empty set. This issue was resolved by retrieving the userID from the 'ratings-personal.csv' file.

The validation process involves two steps. First, it checks whether the specified number of top recommended movies contains a defined set of expected movies. Next, the validation compares the Root Mean Square Error (RMSE) achieved by the best model on the validation subset with the expected value.

The validation does not change the code that is being measured to a greater extent and thus should not really impact benchmark performance. As a quick check, I calculated an average duration from the last 5 repetitions of a single run of the benchmark before and after validation:

without validation: 6916.339ms
(movie-lens.no-validation.result.txt)
with validation: 6973.107ms
(movie-lens.with-validation.result.txt)

The variant with validation appears approx. 1% slower, but this is just from a single run. If necessary, I can do the comparison for multiple runs.

@lbulej lbulej marked this pull request as ready for review April 24, 2024 07:33
@lbulej
Copy link
Member

lbulej commented Apr 24, 2024

LGTM, but I was involved prior to the PR, so I would welcome someone else to take a peek :-)

@lbulej lbulej requested a review from farquet April 24, 2024 08:09
@lbulej lbulej self-assigned this Apr 24, 2024
Copy link
Collaborator

@farquet farquet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Thanks @lovisek for the contribution.

@lbulej lbulej merged commit 65d596e into renaissance-benchmarks:master Apr 29, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants