Replace DvDateTimeZone, DvDateTime, DvTimeSpan with .NET standard types. #693

codemzs · 2018-08-19T07:05:42Z

fixes #673

codemzs · 2018-08-19T15:43:02Z

@TomFinley @eerhardt Do we need missing value support for datetime, timespan, datetimeoffset? These are structs so may be use default as missing values? For now I have assumed there is no missing value indicator but I can add it back. #Resolved

Ivanidzo4ka · 2018-08-19T17:50:32Z

src/Microsoft.ML.Data/DataLoadSave/Binary/CodecFactory.cs

@@ -54,11 +54,11 @@ public CodecFactory(IHostEnvironment env, MemoryStreamPool memPool = null)
            RegisterSimpleCodec(new UnsafeTypeCodec<ulong>(this));
            RegisterSimpleCodec(new UnsafeTypeCodec<Single>(this));
            RegisterSimpleCodec(new UnsafeTypeCodec<Double>(this));
-            RegisterSimpleCodec(new UnsafeTypeCodec<DvTimeSpan>(this));
+            RegisterSimpleCodec(new UnsafeTypeCodec<TimeSpan>(this));


UnsafeTypeCodec [](start = 36, length = 15)

Is it still need to be UnsafeTypeCodec? #Resolved

What should it be? I have defined TimeSpanUnsafeTypeOps to handle this type.

In reply to: 211110052 [](ancestors = 211110052)

TomFinley · 2018-08-20T15:56:34Z

Hi @codemzs not sure I understand, are you proposing that the type for these structures be, for instance, not DateTime but DateTime? given that the default value for DataTime is not sensible?

In reply to: 414136160 [](ancestors = 414136160)

TomFinley · 2018-08-20T15:57:35Z

src/Microsoft.ML/Data/TextLoader.cs

                kind = DataKind.DT;
-            else if (type == typeof(DvDateTimeZone) || type == typeof(TimeZoneInfo))
+            else if (type == typeof(DateTimeOffset) || type == typeof(TimeZoneInfo))


TimeZoneInfo [](start = 70, length = 12)

In what way does a TimeZoneInfo map into a DateTimeOffset? #Resolved

I agree this seems totally wrong.

In reply to: 211315779 [](ancestors = 211315779)

It was a miss, I did intend to remove it.

In reply to: 211662608 [](ancestors = 211662608,211315779)

TomFinley · 2018-08-20T16:00:25Z

src/Microsoft.ML.Core/Data/ColumnType.cs

@@ -135,7 +135,7 @@ public bool IsTimeSpan
        }

        /// <summary>
-        /// Whether this type is a DvDateTime.
+        /// Whether this type is a DateTime.


DateTime [](start = 35, length = 8)

<see tags perhaps #Resolved

TomFinley · 2018-08-20T16:00:26Z

            return true;

I know you didn't write this code but this if (...) return false; return true; type code is incredibly annoying to me...

Maybe:

Contracts.Assert((this == DateTimeType.Instance) == (this is DateTimeType));
return this is DateTimeType

Similar cleanup possible for the other Is properties here. #Resolved

Refers to: src/Microsoft.ML.Core/Data/ColumnType.cs:148 in e0d66b0. [](commit_id = e0d66b0, deletion_comment = False)

TomFinley · 2018-08-20T16:02:27Z

src/Microsoft.ML.Data/Data/Conversion.cs

                return true;
-            }
-            dst = DvDateTime.NA;
+
            return IsStdMissing(ref src);


IsStdMissing(ref src); [](start = 19, length = 22)

So I feel like if you parse missing into a value type that does not support missing, we might expect the result to be that we would throw, rather than just the default value. #Resolved

TomFinley · 2018-08-20T16:02:37Z

src/Microsoft.ML.Data/Data/Conversion.cs

                return true;
-            }
-            dst = DvDateTimeZone.NA;
+
            return IsStdMissing(ref src);


IsStdMissing [](start = 19, length = 12)

Similar for this. #Resolved

TomFinley · 2018-08-20T16:03:58Z

I might have expected to see some change where we take out DvDateTime, Dv... etc. Is it not so? #Resolved

eerhardt · 2018-08-21T16:12:49Z

src/Microsoft.ML.Data/DataLoadSave/Binary/Codecs.cs

            {
-                private List<short> _offsets;


I'm not sure on how concerned we need to be with sizes, but an offset can only be in the range of -14 to 14 hours. So using a long seems like a bit of an overkill.

It also appears that previously the _offsets was the number of minutes in the offset. Now we are using the number of ticks in the offset. Do we need to be concerned with this difference? #Resolved

.NET standard type has offsets in ticks. We could convert offset in minutes when writing to disk and convert it back to ticks when reading but we may lose precision. What do you think?

In reply to: 211667357 [](ancestors = 211667357)

DateTimeOffset exposes its offset as a TimeSpan, but internally it uses short and in minutes.

https://github.com/dotnet/coreclr/blob/9499b08eefd895158c3f3c7834e185a73619128d/src/System.Private.CoreLib/shared/System/DateTimeOffset.cs#L51-L53

https://github.com/dotnet/coreclr/blob/9499b08eefd895158c3f3c7834e185a73619128d/src/System.Private.CoreLib/shared/System/DateTimeOffset.cs#L286-L292

From everything I can find online (ISO8601, RFC3339, SQL Server doc), the offset supports the range -14 to 14 hours, and only supports minute precision.

In reply to: 212488658 [](ancestors = 212488658,211667357)

Thanks a lot! this is helpful. I'm now writing the ticks in minutes as short values because they are already validated when TimeSpan is created and when reading I convert the minutes back into Ticks when creating the TimeSpan object.

In reply to: 212670585 [](ancestors = 212670585,212488658,211667357)

TomFinley · 2018-08-22T18:41:20Z

test/Microsoft.ML.TestFramework/DataPipe/TestDataPipeBase.cs

@@ -906,11 +906,11 @@ protected Func<bool> GetColumnComparer(IRow r1, IRow r2, int col, ColumnType typ
                case DataKind.Bool:
                    return GetComparerOne<DvBool>(r1, r2, col, (x, y) => x.Equals(y));
                case DataKind.TimeSpan:
-                    return GetComparerOne<DvTimeSpan>(r1, r2, col, (x, y) => x.Equals(y));
+                    return GetComparerOne<TimeSpan>(r1, r2, col, (x, y) => x.Ticks == y.Ticks);


TimeSpan [](start = 42, length = 8)

As near as I can see, both TimeSpan and DateTime are directly comparable, that might be better. #Resolved

…to dvdatetime

eerhardt · 2018-08-28T00:12:03Z

src/Microsoft.ML.Data/DataLoadSave/Binary/Codecs.cs

+                    //https://github.com/dotnet/coreclr/blob/9499b08eefd895158c3f3c7834e185a73619128d/src/System.Private.CoreLib/shared/System/DateTimeOffset.cs#L51-L53
+                    //https://github.com/dotnet/coreclr/blob/9499b08eefd895158c3f3c7834e185a73619128d/src/System.Private.CoreLib/shared/System/DateTimeOffset.cs#L286-L292
+                    //From everything online(ISO8601, RFC3339, SQL Server doc, the offset supports the range -14 to 14 hours, and only supports minute precision.
+                    _offsets.Add((short)(value.Offset.Ticks / TimeSpan.TicksPerMinute));


Why can't this just be value.TotalMinutes? #Resolved

don't see totalMinutes under value, did you mean under offset?

In reply to: 213152272 [](ancestors = 213152272)

Yes, value.Offset.TotalMinutes #Resolved

eerhardt · 2018-08-28T00:13:22Z

src/Microsoft.ML.Data/DataLoadSave/Binary/Codecs.cs

                {
                    Contracts.Assert(!_disposed);
-                    value = new DvDateTimeZone(_ticks[_index], _offsets[_index]);
+                    value = new DateTimeOffset(new DateTime(_ticks[_index]), new TimeSpan(_offsets[_index] * TimeSpan.TicksPerMinute));


new TimeSpan(_offsets[_index] * TimeSpan.TicksPerMinute) should be new TimeSpan(0, _offsets[_index], 0) #Resolved

…to dvdatetime

TomFinley · 2018-08-28T14:58:33Z

src/Microsoft.ML.Data/DataLoadSave/Binary/CodecFactory.cs

            RegisterOtherCodec("VBuffer", GetVBufferCodec);
+            RegisterOtherCodec("DvDateTimeZone", new DateTimeOffsetCodec(this).GetCodec);
+            RegisterOtherCodec("DvDateTime", new DateTimeCodec(this).GetCodec);
+            RegisterOtherCodec("DvTimeSpan", new UnsafeTypeCodec<TimeSpan>(this).GetCodec);
            RegisterOtherCodec("Key", GetKeyCodec);


I see the new test helped show up some issues.

BTW where are we testing the writing of data into what amounts to a new format? #Resolved

We have a test that reads from the old IDV format then it writes back as text and that is where writing of data into new format is tested. Also for date/time the serialization format has not changed. I believe it only changed for bools.

In reply to: 213349875 [](ancestors = 213349875)

Maybe a simple way to do this is modify your test a little bit.

Currently it is something like this.

TestCore("savedata", idvPath, "loader=binary", "saver=text", textOutputPath.Arg("dout"));

If we modify it to be this:

var intermediateData = /// some temp IDV file. TestCore("savedata", idvPath, "loader=binary", "saver=text", textOutputPath.Arg("dout")); TestCore("savedata", idvPath, "loader=binary", "saver=binary", intermediateData.ArgOnly("dout")); TestCore("savedata", intermediateData, "loader=binary", "saver=text", textOutputPath.Arg("dout"));

That will test everything. #Resolved

into dvdatetime

…to dvdatetime

…to dvdatetime # Conflicts: # test/Microsoft.ML.TestFramework/DataPipe/TestDataPipeBase.cs

…to dvdatetime

…to dvdatetime # Conflicts: # test/BaselineOutput/SingleDebug/SavePipe/TestParquetPrimitiveDataTypes-Data.txt # test/BaselineOutput/SingleDebug/SavePipe/TestParquetPrimitiveDataTypes-Schema.txt # test/BaselineOutput/SingleRelease/SavePipe/TestParquetPrimitiveDataTypes-Data.txt # test/BaselineOutput/SingleRelease/SavePipe/TestParquetPrimitiveDataTypes-Schema.txt # test/data/Parquet/alltypes.parquet

…to dvdatetime

codemzs · 2018-09-20T18:16:47Z

This change was committed via #863

codemzs added 2 commits August 18, 2018 23:58

Replace date/time types with .NET standard types.

bc530e9

rename DateTimeZone to DateTimeOffset.

e0d66b0

codemzs requested review from TomFinley, dotnet-bot, Ivanidzo4ka, eerhardt and Zruty0 and removed request for dotnet-bot August 19, 2018 07:10

Ivanidzo4ka reviewed Aug 19, 2018

View reviewed changes

TomFinley reviewed Aug 20, 2018

View reviewed changes

eerhardt reviewed Aug 21, 2018

View reviewed changes

TomFinley reviewed Aug 22, 2018

View reviewed changes

codemzs added 4 commits August 26, 2018 17:18

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

54145da

…to dvdatetime

PR feedback.

e55c780

PR feedback.

53a84d1

PR feedback.

b7d4c6b

eerhardt reviewed Aug 28, 2018

View reviewed changes

codemzs added 2 commits August 28, 2018 01:10

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

844da7c

…to dvdatetime

IDV test.

dbaac06

TomFinley reviewed Aug 28, 2018

View reviewed changes

codemzs added 2 commits August 28, 2018 09:29

add test for parquet loader.

53b4e57

Merge branch 'parquettest' of https://github.com/codemzs/machinelearning

ae9aede

into dvdatetime

codemzs added 3 commits August 28, 2018 09:56

Update parquet tests.

48b4c08

fix build.

7f55c59

PR feedback.

71f8701

shauheen assigned codemzs Aug 28, 2018

shauheen added the enhancement New feature or request label Aug 28, 2018

shauheen added this to the 0818 milestone Aug 28, 2018

eerhardt approved these changes Aug 29, 2018

View reviewed changes

codemzs added 2 commits August 28, 2018 21:44

clean up.

1ccc263

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

1672739

…to dvdatetime

shauheen removed this from the 0818 milestone Aug 30, 2018

codemzs added 4 commits August 30, 2018 13:18

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

5231b51

…to dvdatetime

PR feedback.

0a1ba05

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

0560602

…to dvdatetime # Conflicts: # test/Microsoft.ML.TestFramework/DataPipe/TestDataPipeBase.cs

rebuild.

401fa64

codemzs force-pushed the dvdatetime branch from f599ca2 to 401fa64 Compare August 31, 2018 03:27

codemzs added 5 commits August 30, 2018 20:28

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

c3a5a2d

…to dvdatetime

merge parquet test.

7cc7d9a

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

a42861c

…to dvdatetime

merge master.

ea943d6

codemzs closed this Sep 20, 2018

codemzs deleted the dvdatetime branch September 20, 2018 18:16

ghost locked as resolved and limited conversation to collaborators Mar 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace DvDateTimeZone, DvDateTime, DvTimeSpan with .NET standard types. #693

Replace DvDateTimeZone, DvDateTime, DvTimeSpan with .NET standard types. #693

codemzs commented Aug 19, 2018 •

edited

Loading

codemzs commented Aug 19, 2018 •

edited

Loading

Ivanidzo4ka Aug 19, 2018 •

edited by codemzs

Loading

codemzs Aug 27, 2018

TomFinley commented Aug 20, 2018

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

eerhardt Aug 21, 2018

codemzs Aug 27, 2018

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley commented Aug 20, 2018 •

edited by codemzs

Loading

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley commented Aug 20, 2018 •

edited by codemzs

Loading

eerhardt Aug 21, 2018 •

edited by codemzs

Loading

codemzs Aug 23, 2018 •

edited

Loading

eerhardt Aug 24, 2018

codemzs Aug 27, 2018

TomFinley Aug 22, 2018 •

edited by codemzs

Loading

eerhardt Aug 28, 2018 •

edited by codemzs

Loading

codemzs Aug 28, 2018

eerhardt Aug 28, 2018 •

edited by codemzs

Loading

eerhardt Aug 28, 2018 •

edited by codemzs

Loading

TomFinley Aug 28, 2018 •

edited by codemzs

Loading

codemzs Aug 28, 2018 •

edited

Loading

TomFinley Aug 29, 2018 •

edited by codemzs

Loading

codemzs commented Sep 20, 2018

Replace DvDateTimeZone, DvDateTime, DvTimeSpan with .NET standard types. #693

Replace DvDateTimeZone, DvDateTime, DvTimeSpan with .NET standard types. #693

Conversation

codemzs commented Aug 19, 2018 • edited Loading

codemzs commented Aug 19, 2018 • edited Loading

Ivanidzo4ka Aug 19, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

codemzs Aug 27, 2018

Choose a reason for hiding this comment

TomFinley commented Aug 20, 2018

TomFinley Aug 20, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

eerhardt Aug 21, 2018

Choose a reason for hiding this comment

codemzs Aug 27, 2018

Choose a reason for hiding this comment

TomFinley Aug 20, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

TomFinley commented Aug 20, 2018 • edited by codemzs Loading

TomFinley Aug 20, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

TomFinley Aug 20, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

TomFinley commented Aug 20, 2018 • edited by codemzs Loading

eerhardt Aug 21, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

codemzs Aug 23, 2018 • edited Loading

Choose a reason for hiding this comment

eerhardt Aug 24, 2018

Choose a reason for hiding this comment

codemzs Aug 27, 2018

Choose a reason for hiding this comment

TomFinley Aug 22, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

eerhardt Aug 28, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

codemzs Aug 28, 2018

Choose a reason for hiding this comment

eerhardt Aug 28, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

eerhardt Aug 28, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

TomFinley Aug 28, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

codemzs Aug 28, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley Aug 29, 2018 • edited by codemzs Loading

Choose a reason for hiding this comment

codemzs commented Sep 20, 2018

codemzs commented Aug 19, 2018 •

edited

Loading

codemzs commented Aug 19, 2018 •

edited

Loading

Ivanidzo4ka Aug 19, 2018 •

edited by codemzs

Loading

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley commented Aug 20, 2018 •

edited by codemzs

Loading

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley Aug 20, 2018 •

edited by codemzs

Loading

TomFinley commented Aug 20, 2018 •

edited by codemzs

Loading

eerhardt Aug 21, 2018 •

edited by codemzs

Loading

codemzs Aug 23, 2018 •

edited

Loading

TomFinley Aug 22, 2018 •

edited by codemzs

Loading

eerhardt Aug 28, 2018 •

edited by codemzs

Loading

eerhardt Aug 28, 2018 •

edited by codemzs

Loading

eerhardt Aug 28, 2018 •

edited by codemzs

Loading

TomFinley Aug 28, 2018 •

edited by codemzs

Loading

codemzs Aug 28, 2018 •

edited

Loading

TomFinley Aug 29, 2018 •

edited by codemzs

Loading