[DNN Training] Failed save model in Microsoft.ML.Transforms.DnnTransformer.SaveModel

[yamachu]

System information

• OS version/distro: maxOS 10.14.6
• .NET Version (eg., dotnet --info): 3.0.100-preview9-014004

Issue

• What did you do?

  • Use RetrainDnnModel for training regression-based deep-neural-network model
  • Try to save trained model

• What happened?

Application throw unhandled exception: System.ArgumentNullException and crash

• What did you expect?

We could save trained model

Source code / logs

Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'value')
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.ML.Transforms.DnnTransformer.SaveModel(ModelSaveContext ctx)
   at Microsoft.ML.Data.RowToRowTransformerBase.Microsoft.ML.ICanSaveModel.Save(ModelSaveContext ctx)
   at Microsoft.ML.ModelSaveContext.SaveModel[T](RepositoryWriter rep, T value, String path)
   at Microsoft.ML.ModelOperationsCatalog.Save(ITransformer model, DataViewSchema inputSchema, Stream stream)
   at Microsoft.ML.ModelOperationsCatalog.Save(ITransformer model, DataViewSchema inputSchema, String filePath)
   at VoiceConversionStarter.Console.Program.Train(TrainMcapOptions opts) in /Users/yamachu/Projects/github.com/yamachu/VoiceConversionStarter/VoiceConversionStarter.Console/Program.cs:line 100
   at VoiceConversionStarter.Console.Program.<>c.<Main>b__2_0(TrainMcapOptions opts) in /Users/yamachu/Projects/github.com/yamachu/VoiceConversionStarter/VoiceConversionStarter.Console/Program.cs:line 118
   at CommandLine.ParserResultExtensions.MapResult[TSource,TResult](ParserResult`1 result, Func`2 parsedFunc, Func`2 notParsedFunc)
   at VoiceConversionStarter.Console.Program.Main(String[] args) in /Users/yamachu/Projects/github.com/yamachu/VoiceConversionStarter/VoiceConversionStarter.Console/Program.cs:line 116

I think this issue is not to check null in this line

machinelearning/src/Microsoft.ML.Dnn/DnnTransform.cs

Lines 1118 to 1119 in 8d51eee

	ctx.Writer.Write(_predictionTensorName);
	ctx.Writer.Write(_softmaxTensorName);

That field in my trained model is null

full source code: https://github.com/yamachu/VoiceConversionStarter/tree/pkg-update

[codemzs]

@yamachu can you please try latest nuget? 0.16-preview

[yamachu]

@codemzs Perfect! It works!

env:

• Microsoft.ML{,.TensorFlow} 1.4.0-preview
• Microsoft.ML.Dnn 0.16.0-preview

[yamachu]

Model saving works now.
But, I cannot load that model...

logs

Unhandled exception. System.InvalidOperationException: Error during class instantiation
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.FormatException: One of the identified items was in an invalid format.
   at Microsoft.ML.Transforms.DnnRetrainTransformer.Create(IHostEnvironment env, ModelLoadContext ctx)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstance(IHostEnvironment env, Object args, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at VoiceConversionStarter.Console.Program.Eval(EvalMcapOptions opts) in /Users/yamachu/Projects/github.com/yamachu/VoiceConversionStarter/VoiceConversionStarter.Console/Program.cs:line 145
   at VoiceConversionStarter.Console.Program.<>c.<Main>b__4_1(EvalMcapOptions opts) in /Users/yamachu/Projects/github.com/yamachu/VoiceConversionStarter/VoiceConversionStarter.Console/Program.cs:line 171
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 notParsedFunc)
   at VoiceConversionStarter.Console.Program.Main(String[] args) in /Users/yamachu/Projects/github.com/yamachu/VoiceConversionStarter/VoiceConversionStarter.Console/Program.cs:line 168