Updating DatabaseLoader to support getting column info from a given .NET type. #4091
Conversation
src/Microsoft.ML.Experimental/DataLoadSave/Database/DatabaseLoader.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.ML.Experimental/DataLoadSave/Database/DatabaseLoader.cs
Outdated
Show resolved
Hide resolved
| return DbType.Boolean; | ||
| } | ||
|
|
||
| case InternalDataKind.DT: |
There was a problem hiding this comment.
What about DateTimeOffset (InternalDataKind.DZ) and TimeSpan (InternalDataKind.TS) ?
There was a problem hiding this comment.
There is no equivalent for DbKind
There was a problem hiding this comment.
Even DbType.DateTimeOffset and DbType.Time?
https://docs.microsoft.com/en-us/dotnet/api/system.data.dbtype
There was a problem hiding this comment.
The latter is the SQL time type, not an equivalent to the System.TimeSpan type; I'm not sure what the appropriate conversion is. I missed DateTimeOffset.
There was a problem hiding this comment.
The latter is the SQL time type, not an equivalent to the System.TimeSpan type;
According to https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-data-type-mappings, it is the appropriate mapping:
| SQL Server Database Engine type | .NET Framework type | SqlDbType enumeration | SqlDataReader SqlTypes typed accessor | DbType enumeration | SqlDataReader DbType typed accessor |
|---|---|---|---|---|---|
| time(SQL Server 2008 and later) | TimeSpan | Time | none | Time | GetDateTime |
There was a problem hiding this comment.
Is that also the appropriate mapping for all database types, not just SQL?
Also, it looks like it isn't even the SQL time type, it is the SQL datetime type and there is a separate value in a separate enum that should be used for SQL time:
| Time | 17 | A type representing a SQL Server DateTime value. If you want to use a SQL Server time value, use Time. |
There was a problem hiding this comment.
ah, right.
These weren't done because DbDataReader doesn't expose Get methods for them
src/Microsoft.ML.Experimental/DataLoadSave/Database/DatabaseLoaderCatalog.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.ML.Experimental/DataLoadSave/Database/DatabaseLoader.cs
Outdated
Show resolved
Hide resolved
| public int? Max; | ||
|
|
||
| /// <summary> | ||
| /// Whether this range extends to the end of the line, but should be a fixed number of items. |
There was a problem hiding this comment.
What does "to the end of the line" mean to a database loader?
I think we should remove these options until we decide we have data that says they are required. That way they don't "sneak" in to the product, and don't do anything.
There was a problem hiding this comment.
In this case it would mean to the end of the table. I'm pretty sure it is used for vector support.
|
@eerhardt, any other feedback here or can it be merged and I can start on getting the vector support up? |
I don't believe that this feedback has been addressed:
We shouldn't be adding public options that aren't being used. You can easily add these new options if and when they get supported. |
They are going to be used in the next PR which adds the vector support. Is it really worthwhile to pull it out and retrigger CI just for it to be used and re-added in the following PR? |
ghost
locked as resolved and limited conversation to collaborators
This adds limited support for determining the column info from a .NET type in order to match what TextLoader provides.