-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ITrialResultManager for continue training in AutoML #6335
Add ITrialResultManager for continue training in AutoML #6335
Conversation
@@ -12,13 +12,13 @@ namespace Microsoft.ML.AutoML | |||
public interface IMetricManager | |||
{ | |||
bool IsMaximize { get; } | |||
|
|||
string MetricName { get; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"loss", | ||
"durationInMilliseconds", | ||
"peakCpu", | ||
"peakMemoryInMegaByte" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: please align this block
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved
{ | ||
if (!File.Exists(filePath)) | ||
{ | ||
return new TrialResult[0]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved
@@ -56,6 +54,11 @@ public Flow2(SearchSpace.SearchSpace searchSpace, Parameter initValue = null, bo | |||
{ | |||
_step = _stepUpperBound; | |||
} | |||
|
|||
if (rng != null) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved
{ | ||
// initialize eci with the estimated cost and always start from pipeline which has lowest cost. | ||
_eci = _pipelineSchemas.ToDictionary(kv => kv, kv => GetEstimatedCostForPipeline(kv, _sweepablePipeline)); | ||
_initialized = true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved
} | ||
|
||
public void Update(TrialResult result, string schema) | ||
{ | ||
_initialized = true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved
} | ||
|
||
// make sure hash code is greater than 0 | ||
return (int)(hash >> 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it picked up from Java? :-)
You can simplify that like
https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/Tuple.cs,69
without using checked blocks at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's cool, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added minor comments and questions. LGTM otherwise.
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6335 +/- ##
==========================================
- Coverage 68.58% 68.51% -0.08%
==========================================
Files 1171 1165 -6
Lines 247964 246142 -1822
Branches 25738 25555 -183
==========================================
- Hits 170076 168635 -1441
+ Misses 71129 70834 -295
+ Partials 6759 6673 -86
Flags with carried forward coverage won't be shown. Click here to find out more.
|
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
disable SweepablePipeline_AutoFit_UCI_Adult_CrossValidation_10_Test
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp run |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
/azp --help |
Command '--help' is not supported by Azure Pipelines. Supported commands
See additional documentation. |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
We are excited to review your PR.
So we can do the best job, please check:
Fixes #nnnn
in your description to cause GitHub to automatically close the issue(s) when your PR is merged.Resolved issue
#5736
What's
ITrialResultManager
ITrialResultManager
is a component used byAutoMLExperiment
to save and loadTrialResult
. The default implementation forITrialResultManager
isCsvTrialResultManager
, which saves all trial results in csv formatWhat's
ITrialResultManager
used forCurrently, it's used by some tuners (CostFrugalTuner, for example) to recover from previous training, suppose trialResult csv is provided or is available.
How to use it
ITrialResultManager
is not directly exposed to public users. User can useSetCheckpoint(string folder)
path instead. If a folder is provided viaSetCheckpoint
,AutoMLExperiment
will create that folder if not exists, and configuresCsvTrialResultManager
to save trial results tofolder/trialResults.csv
.