Date conversion returns unintuitive output

[jeremiahpslewis]

In Julia 0.3.6; x86_64-apple-darwin13.4.0

julia> Date("01/01/05", "mm/dd/yy")
0005-01-01

[quinnj]

We take a conservative approach in parsing that doesn't try to guess what century your date is in. We could possibly have a YY formatting option that would signify 2-digit years, but even then, we have to define some kind of split for when we assume 1900 vs. 2000. With the current setup, we leave that in in the user's hands to do

Date("01/01/05", "mm/dd/yy") + Dates.Year(2000)

[jeremiahpslewis]

Makes sense. Probably one of those things that make for good language design, but also life slightly more difficult when one is switching from a language like R (which parses the string more reasonably & where the cutoff is between 68 and 69). Still, the current behavior is kind of absurd...it's no one's reasonable guess that the date refers to year 5. Would it be possible for Date("01/01/05", "mm/dd/yy") to throw an exception and required a base year or something?

[JeffBezanson]

According to wikipedia, ISO-8601 says

ISO 8601 prescribes, as a minimum, a four-digit year [YYYY] to avoid the year 2000 problem. It therefore represents years from 0000 to 9999, year 0000 being equal to 1 BC and all others AD. However, years prior to 1583 are not automatically allowed by the standard. Instead "values in the range [0000] through [1582] shall only be used by mutual agreement of the partners in information interchange."

So perhaps a compromise is to give an error for years prior to 1583?

[ivarne]

I also think the current behavior is absurd. It's pretty uncommon to parse dates before year 100 (and do we even support negative years?). Most 2 digit years will be from the last (or next) 50 25 years.

[nalimilan]

The difficulty is to find a reasonable cut year to choose between +1900 and +2000. Maybe 50 years after vs. 50 years before the current year? I.e. for 2015 values below 65 would be added 2000, and those above would be added 1900?

Note that the + Dates.Year(2000) trick does not work if you have 99 and 02 meaning 1999 and 2002, as is very common. So even if nothing happens automatically, there needs to be at least an argument to specify the cut year.

[JeffBezanson]

It would be ok for parsing a 2-digit year to be an error if a pivot year is not specified.

[johnmyleswhite]

+1 for raising an error in response to ambiguity

[mbauman]

Maybe 50 years after vs. 50 years before the current year?

Yikes, that's just as bad, or perhaps even worse. I come back to work after christmas break and suddenly the script that was working bombs out because I have a date from 1965 in my dataset. +1 for an error here, especially if we add the ability to specify an arbitrary pivot year.

[ivarne]

All 2 digit years are ambiguous, so do you think we should refuse to parse the format string without a manually specified pivot?

If you have more than 20 years time span in your data, you should really use 4 digit year, as any default we can come up with has potential new years bombs in them.

[JeffBezanson]

Yes I think we should refuse 2-digit years without a manually specified pivot. No standard date formats support 2-digit years as far as I can tell.

[ivarne]

OK, then we need to find a syntax for a date parsing string with a pivot.

[mbauman]

We've been calling it a pivot here, but it's really the minimum year. Perhaps a keyword argument minyear? Then even if you have a 4-digit year format string with a minyear, it will error if any year is less than that.

Alternatively, we could just make it undefined behavior and shout at users to not use it in the help:

 '%y' Year without century (00-99). If you use this on input, which
      century you get is system-specific.  So don't!  Often values
      up to 69 (or 68) are prefixed by 20 and 70 (or 69) to 99 by
      19.

[ivarne]

# parse date between 1. January 1940 and 31. December 2039
Date("01/01/05", "mm/dd/y1940y")
# parse date less than 50 years to the future
Date("01/01/05", "mm/dd/y50y")
# parse a 2 digit year in the past (eg. log files)
julia> Date("01/01/05", "mm/dd/y0y")

[nalimilan]

I would find this easier to understand: Date("01/01/05", "mm/dd/yy", pivot=1940) or Date("01/01/05", "mm/dd/yy", minyear=1940).

[quinnj]

I'm definitely more in favor of the keyword argument. So with minyear, the usage would be:

Date("01/01/05", "mm/dd/yy", minyear=1940) # returns 2005 because next valid '05' year after 1940
Date("01/01/05", "mm/dd/yy", minyear=1900) # returns 1905

Date("01/01/99", "mm/dd/yy", minyear=1940) # returns 1999

Date("01/01/39", "mm/dd/yy", minyear=1940) # returns 2039
Date("01/01/40", "mm/dd/yy", minyear=1940) # returns 1940

I think I like this because the limitation is Date("01/01/05", "mm/dd/yy", minyear=1900) can never parse to 2005, which is accurate since at that point the ambiguity enters. Specifying the minyear effectively defines the starting year of your "century" of valid two-digit years.

Now, a question on the implementation. Typically this would be a case for a "required keyword" argument like:

Date(datestring, formatstring; minyear::Int=error("must specify minyear"))

yet we only want to throw the error when a two-digit year is encountered. Thoughts on the best way to approach this? Use a Nullable for the keyword argument default value?

function Date(datestring, formatstring; minyear::Int=Nullable{Int}())
    if encountered_two_digit_year()
        isnull(minyear) && error("must specify minyear keyword argument for parsing two-digit years")
...