Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get Loss During Training for Visualization (Learning Curve Graph) #7140

Open
AutumnEvans418 opened this issue Apr 23, 2024 · 0 comments
Open
Labels
enhancement New feature or request Evaluation Issues related to evaluation untriaged New issue has not been triaged

Comments

@AutumnEvans418
Copy link

Is your feature request related to a problem? Please describe.
I need a way to visualize how my model is learning during training, which is a comparison between training loss and test loss.

Describe the solution you'd like
An event handler that enables the ability to extract loss during training.

Describe alternatives you've considered
Running the model for x epochs, evaluating the model, then retraining the model in a loop. This unfortunately does not work for all models, such as LightGbm that can't be retrained.

var kfold = ctx.BinaryClassification.CrossValidate(training, estimator, param.kfold);
var bestModel = kfold.OrderByDescending(p => p.Metrics.Accuracy).Select(p => p.Model).First();

var testOutput = bestModel.Transform(test);

var metrics = ctx.BinaryClassification.Evaluate(testOutput);

// This code doesn't work as estimator is IEstimator<ITransform> and bestModel is ITransform. Not sure how you would do this...
estimator = bestModel;

Additional context
Ultimately, I am trying to analyze what the models I'm comparing are actually doing and so far I haven't found any documentation or any straightforward way to do it.
image

@AutumnEvans418 AutumnEvans418 added the enhancement New feature or request label Apr 23, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged label Apr 23, 2024
@luisquintanilla luisquintanilla added the Evaluation Issues related to evaluation label Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Evaluation Issues related to evaluation untriaged New issue has not been triaged
Projects
None yet
Development

No branches or pull requests

2 participants