CreateEnumerable code fails in Visual Studio FSI but works in .NET Interactive notebook?

[nhirschey]

System Information (please complete the following information):

• OS & Version: Windows 11
• ML.NET Version: Microsoft.ML.AutoML, 0.20.0-preview.22356.1
• .NET Version: failed on both 6.0.400 and 7.0.100-rc.1.22431.12

Describe the bug
I am trying to create an F# IEnumerable from predictions. The code works in .NET Interactive notebooks, but it does not work in Visual Studio 2022 17.3.2 via F# interactive. I have also tried pasting the code into F# interactive by running dotnet fsi at the command line, but I get the same error.

When I run it in a notebook, I get predictions as expected. When I run it in a FSX script file in Visual Studio, I get the error "Could not find column 'Predicted@' (Parameter 'Schema')", yet the schema has a parameter named "Predicted".

To Reproduce
Code

#i "nuget:https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json"
#r "nuget: Microsoft.ML.AutoML, 0.20.0-preview.22356.1"

open System
open Microsoft.ML
open Microsoft.ML.Data

let mlContext = new MLContext()

[<CLIMutable>]
type Predictors =
    {
        Date: DateTime
        Return: single
        DividendPrice: single
    }

let predictors =
    [{ Date = DateTime(1872,2,1)
       Return = 0.008688271977f
       DividendPrice = 0.05482897535f };
     { Date = DateTime(1872,3,1)
       Return = 0.03739754111f
       DividendPrice = 0.05333504081f };
     { Date = DateTime(1872,4,1)
       Return = 0.03229662776f
       DividendPrice = 0.05175264552f};
     { Date = DateTime(1872,5,1)
       Return = 0.004451415502f
       DividendPrice = 0.05051480234f }]

let mlData = mlContext.Data.LoadFromEnumerable(predictors)

let modelFit =
    let model =
        EstimatorChain()
            .Append(mlContext.Transforms.CopyColumns("Label","Return"))
            .Append(mlContext.Transforms.Concatenate("Features","DividendPrice"))
            .Append(mlContext.Regression.Trainers.Ols())
    model.Fit(mlData)

[<CLIMutable>]
type Prediction =
    {
        Predicted: single
        Date: DateTime
    }

let pred = modelFit.Transform(mlData)
let renamePipeline = mlContext.Transforms.CopyColumns("Predicted","Score")
let renamed = renamePipeline.Fit(pred).Transform(pred)
mlContext.Data.CreateEnumerable<Prediction>(renamed,reuseRowObject=false)

Output in .NET interactive:

index	Predicted	Date
0	0.022715298	1872-02-01 00:00:00Z
1	0.021365488	1872-03-01 00:00:00Z
2	0.019935748	1872-04-01 00:00:00Z
3	0.018817322	1872-05-01 00:00:00Z

Output in Visual Studio:

System.ArgumentOutOfRangeException: Could not find  column 'Predicted@' (Parameter 'Schema')
   at Microsoft.ML.Data.TypedCursorable`1..ctor(IHostEnvironment env, IDataView data, Boolean ignoreMissingColumns, InternalSchemaDefinition schemaDefn)
   at Microsoft.ML.Data.TypedCursorable`1.Create(IHostEnvironment env, IDataView data, Boolean ignoreMissingColumns, SchemaDefinition schemaDefinition)
   at Microsoft.ML.Data.CursoringUtils.AsCursorable[TRow](IHostEnvironment env, IDataView data, Boolean ignoreMissingColumns, SchemaDefinition schemaDefinition)
   at Microsoft.ML.PipeEngine`1..ctor(IHostEnvironment env, IDataView pipe, Boolean ignoreMissingColumns, SchemaDefinition schemaDefinition)
   at Microsoft.ML.DataOperationsCatalog.CreateEnumerable[TRow](IDataView data, Boolean reuseRowObject, Boolean ignoreMissingColumns, SchemaDefinition schemaDefinition)
   at <StartupCode$FSI_0004>.$FSI_0004.main@() in C:\test.fsx:line 52
Stopped due to error

Here is the schema

> renamed.Schema |> Seq.toArray
val it: DataViewSchema.Column[] =
  [|Date@: DateTime {Annotations = ;
                     Index = 0;
                     IsHidden = false;
                     Name = "Date@";
                     Type = DateTime;}; Return@: Single {Annotations = ;
                                                         Index = 1;
                                                         IsHidden = false;
                                                         Name = "Return@";
                                                         Type = Single;};
    DividendPrice@: Single {Annotations = ;
                            Index = 2;
                            IsHidden = false;
                            Name = "DividendPrice@";
                            Type = Single;}; Date: DateTime {Annotations = ;
                                                             Index = 3;
                                                             IsHidden = false;
                                                             Name = "Date";
                                                             Type = DateTime;};
    Return: Single {Annotations = ;
                    Index = 4;
                    IsHidden = false;
                    Name = "Return";
                    Type = Single;};
    DividendPrice: Single {Annotations = ;
                           Index = 5;
                           IsHidden = false;
                           Name = "DividendPrice";
                           Type = Single;}; Label: Single {Annotations = ;
                                                           Index = 6;
                                                           IsHidden = false;
                                                           Name = "Label";
                                                           Type = Single;};
    Features: Vector<Single, 1> {SlotNames} {Annotations = SlotNames;
                                             Index = 7;
                                             IsHidden = false;
                                             Name = "Features";
                                             Type = Vector<Single, 1>;};
    Score: Single {ScoreColumnSetId, ScoreColumnKind, ScoreValueKind}
      {Annotations = ScoreColumnSetId, ScoreColumnKind, ScoreValueKind;
       Index = 8;
       IsHidden = false;
       Name = "Score";
       Type = Single;};
    Predicted: Single {ScoreColumnSetId, ScoreColumnKind, ScoreValueKind}
      {Annotations = ScoreColumnSetId, ScoreColumnKind, ScoreValueKind;
       Index = 9;
       IsHidden = false;
       Name = "Predicted";
       Type = Single;}|]

Expected behavior
I expect the code to work in fsharp interactive

[michaelgsharp]

@luisquintanilla any thoughts on this?

[luisquintanilla]

Nothing comes to mind off the top of my head but I can try and repro.

@nhirschey is VS a requirement? If not, I would recommend as a potential workaround using FSI in VS Code.

[nhirschey]

VS is not a requirement, but I get it in every version of FSI that I’ve tried (VS, VSCode with and without Ionide, and starting manually in windows terminal prompt).

I'm grateful for any tips! I'm probably doing something really silly to cause the error, but I cannot figure it out. I'm stumped about the code working in VSCode notebooks but not any version of dotnet fsi.

[luisquintanilla]

This is weird...

I am seeing the same error. I also tried:

• Downgrading versions of AutoML as well as
• Using Microsoft.ML and the Microsoft.ML.MKLImports packages directly

What's odd is, as the error suggests:

System.ArgumentOutOfRangeException: Could not find column 'Predicted@'

The column has an @ suffix in the column name. However if you were to print out the column names or display the schema as you have @nhirschey, the suffix isn't there.

renamed.Schema 
    |> Seq.iter(fun x -> printfn $"{x.Name}")

// outputs
Date@
Return@
DividendPrice@
Date
Return
DividendPrice
Label
Features
Score
Predicted

If you try to get the column using the GetColumn method using the correct name (without the suffix), it works.

renamed.GetColumn<single>("Predicted")

//outputs
seq [0.02271529846f; 0.02136548795f; 0.01993574761f; 0.01881732233f]

Taking into account I tested the code on an F# console app and you said it works on .NET notebooks, it leads me to believe this is not an ML.NET issue, but FSI.

What I would recommend if you want to use this in the context of an interactive environment like FSI is to use DataFrames. You can call ToDataFrame on the IDataView. As a bonus, you don't have to create a separate type for your predictions 🙂.

Here's the code for that:

#i "nuget:https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json"

#r "nuget: Microsoft.ML.AutoML, 0.20.0-preview.22470.1"
#r "nuget: Microsoft.Data.Analysis, 0.20.0-preview.22470.1"

open System
open Microsoft.ML
open Microsoft.ML.Data
open Microsoft.Data.Analysis

let mlContext = new MLContext()

[<CLIMutable>]
type Predictors =
    {
        Date: DateTime
        Return: single
        DividendPrice: single
    }

let predictors =
    [{ Date = DateTime(1872,2,1)
       Return = 0.008688271977f
       DividendPrice = 0.05482897535f };
     { Date = DateTime(1872,3,1)
       Return = 0.03739754111f
       DividendPrice = 0.05333504081f };
     { Date = DateTime(1872,4,1)
       Return = 0.03229662776f
       DividendPrice = 0.05175264552f};
     { Date = DateTime(1872,5,1)
       Return = 0.004451415502f
       DividendPrice = 0.05051480234f }]

let mlData = mlContext.Data.LoadFromEnumerable(predictors)

let modelFit =
    let model =
        EstimatorChain()
            .Append(mlContext.Transforms.CopyColumns("Label","Return"))
            .Append(mlContext.Transforms.Concatenate("Features","DividendPrice"))
            .Append(mlContext.Regression.Trainers.Ols())
    model.Fit(mlData)

let pred = modelFit.Transform(mlData)

let renamePipeline = mlContext.Transforms.CopyColumns("Predicted","Score")
let renamed = renamePipeline.Fit(pred).Transform(pred)

renamed.ToDataFrame(selectColumns=[|"Predicted"; "Date"|])

The resulting output looks like the following:

Date	Predicted
2/1/1872 12:00:00 AM	0.022715298
3/1/1872 12:00:00 AM	0.021365488
4/1/1872 12:00:00 AM	0.019935748
5/1/1872 12:00:00 AM	0.018817322

[nhirschey]

Thank you for the workaround! I will try using DataFrame.

One thought: Is this a reflection issue? At each point in the code where ML.NET is grabbing a column name from an F# type (via reflection?), it seems to be appending "@" to the name.

.NET Interactive may be overriding some .NET Runtime reflection methods to make variable sharing between C# and F# kernels easier. That would explain why ML.NET gets the correct name in a notebook--it could be piggybacking on .NET Interactive reflection methods.

Some evidence, the IDataView schema is different depending on whether you use regular .NET or .NET Interactive Notebooks in VSCode.

.NET FSI

We see that there's two versions of each variable, one with the "NAME@" and one with "NAME"

> let mlData = mlContext.Data.LoadFromEnumerable(predictors)
> mlData.Schema |> Seq.iter(fun x -> printfn $"{x.Name}");;
Date@
Return@
DividendPrice@
Date
Return
DividendPrice
val it: unit = ()

>

.NET Interactive

The IDataView Schema exactly matches the F# record, only "NAME". No "NAME@".

[nhirschey]

Ok, I'm guessing it's how ML.NET accesses F# record field names via reflection. Below is in VSCode fsi.

First, .NET Runtime reflection (what I presume ML.NET is using). It shows field names with "@" appended.

open System
open System.Reflection

[<CLIMutable>]
type Predictors =
    {
        Date: DateTime
        Return: single
        DividendPrice: single
    }

let ty = typeof<Predictors>
ty.GetFields() |> Array.map (fun x -> x.Name)

// val it: string[] = [|"Date@"; "Return@"; "DividendPrice@"|]

Next, Reflection using the F# Reflection for types (.NET Interactive is maybe using this for F# types?). It does not append @ to field names.

Reflection.FSharpType.GetRecordFields(typeof<Predictors>) |> Array.map (fun x -> x.Name);;
// val it: string[] = [|"Date"; "Return"; "DividendPrice"|]

[luisquintanilla]

Thanks for looking deeper into this @nhirschey

I haven't looked deep into this PR but I believe this is when support for records was added.

#616

[nhirschey]

Thanks for the tip @luisquintanilla. Reading the PR and comments, for reflection on F# records ML.NET needs to check the type's PropertyInfo rather than FieldInfo. The #616 added switch statements to achieve that.

Indeed, as the code sample below shows, if you look at the property info you get the name without the "@". So likely something in the CreateEnumerable code path is looking at FieldInfo rather than using a switch statement to also check property info.

I did a brief look at the ML.NET code base to try to track down the problem code but I didn't find it.

open System
open System.Reflection

[<CLIMutable>]
type Predictors =
    {
        Date: DateTime
        Return: single
        DividendPrice: single
    }

let ty = typeof<Predictors>

ty.GetProperties() 
|> Array.map (fun x -> x.Name)
// val it: string[] = [|"Date"; "Return"; "DividendPrice"|]