Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image classification needs cancel #4632

Closed
LittleLittleCloud opened this issue Jan 6, 2020 · 7 comments · Fixed by #4650
Closed

image classification needs cancel #4632

LittleLittleCloud opened this issue Jan 6, 2020 · 7 comments · Fixed by #4650
Assignees
Labels
enhancement New feature or request P2 Priority of the issue for triage purpose: Needs to be fixed at some point.

Comments

@LittleLittleCloud
Copy link
Contributor

LittleLittleCloud commented Jan 6, 2020

Hi, is there any plan to add cancel support for image classification API so that we can cancel an image classification training by passing in a cancellation token?

related issue

@antoniovs1029 antoniovs1029 self-assigned this Jan 6, 2020
@antoniovs1029
Copy link
Member

Hi @LittleLittleCloud , so it's still not clear to me what is being asked, and so if it would be a quick fix or a major feature request.

Can you please share code (and perhaps a dataset) where you "cancel and switch to another scenario" (as mentioned in the issue you've linked to)? I would like to know how you're currently cancelling the training.

Particularly, it's strange that you mention that there's a lot of memory being used and that's not released after cancelling. Perhaps you are pointing to a memory leak?

Thanks.

@LittleLittleCloud
Copy link
Contributor Author

LittleLittleCloud commented Jan 7, 2020

It's not a memory leak, the thing is our program allows user to cancel training after it start, But we can't stop an image training after it starts because current Image Classification API doesn't support cancellation token.

The way we use Image Classification is through AutoML, there's a trainer in MulticlassificationExperiment , and that's how it be called

@codemzs
Copy link
Member

codemzs commented Jan 7, 2020

This should be an easy fix. Please refer to the below PRs where we have added cancellation mechanism in other trainers. You can invoke cancellation via mlcontext cancel function that is exposed in experimental nuget. First, someone needs go add cancellation checkpoints, test, verify with model builder, commit, ship.

https://github.com/dotnet/machinelearning/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+cancellation++

@antoniovs1029
Copy link
Member

antoniovs1029 commented Jan 7, 2020

Would doing what @codemzs has just said (to expose cancelation via mlContext.CancelExecution()) be enough for you, @LittleLittleCloud ?

@LittleLittleCloud
Copy link
Contributor Author

If we can cancel using CancelExecution() that also works. ( As long as there's a way to cancel training)

@codemzs
Copy link
Member

codemzs commented Jan 8, 2020

Yep, we will need to add a check point in the training loop, it’s just few lines of change plus testing.

@antoniovs1029
Copy link
Member

Ok, so I will work on this. Thanks for your suggestions, @codemzs !

@harishsk harishsk added the P1 Priority of the issue for triage purpose: Needs to be fixed soon. label Jan 9, 2020
@ganik ganik added P2 Priority of the issue for triage purpose: Needs to be fixed at some point. and removed P1 Priority of the issue for triage purpose: Needs to be fixed soon. labels Jan 10, 2020
@antoniovs1029 antoniovs1029 added the enhancement New feature or request label Jan 10, 2020
antoniovs1029 added a commit that referenced this issue Jan 17, 2020
* Added CheckAlive() Checkpoints in ImageClassificationTrainer
@ghost ghost locked as resolved and limited conversation to collaborators Mar 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request P2 Priority of the issue for triage purpose: Needs to be fixed at some point.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants