Datetime handling before Unix epoch

[andybrnr]

Requesting daily PX_LAST for the SPX Index from 1927 to present correctly pulls the daily values, but all timestamps before the Unix epoch report as 1969-12-31 23:59:59. I'm pretty sure this has something to do with the difference between the datetime as handled in the C++ API vs in the JavaScript libs, but I'm enough of a n00b with this stuff that I can't quite pin it down.

[ericvw]

@andybrnr, thanks for reporting this issue. Can you provide the precise HTTP request you are making that replicates the issue? Thanks.

[andybrnr]

Hi Eric,

Python 3 code I'm using for the request below. I've had no problems with dates after the Unix Epoch, and the standard Python BLPAPI handles pre-epoch dates correctly (haven't tried to replicate this using only blpapi-node yet).

import json
import urllib.request as urllib

histData = {"securities": ['SPX Index'],
            "fields": ['PX_LAST'],
            "startDate": '19270101',
            "endDate": '20150818',
            "periodicitySelection": 'DAILY'}
data = json.dumps(histData)
bin_data = data.encode('utf8')
clen = len(bin_data)
req = urllib.Request('http://{}/request?ns=blp&service=refdata&type=ReferenceDataRequest'.format(host),
                      bin_data,{'Content-Type': 'application/json', 'Content-Length': clen})

[ericvw]

@andybrnr, this example errors when I try to replicate it similarly in blpapi-node. I don't believe that ReferenceDataRequest's accept startDate, endDate, nor periodicitySelection. Did you intend for this example to be a HistoricalDataRequest?

[ericvw]

If I switch to HistoricalDataRequest, I am unable to replicate any timestamps that are equal to "1969-12-31 23:59:59". In blpapi-node, I modified one of the .js examples and the timestamps returned appeared correct. Additionally, for blpapi-http, I modified one of the shell/curl examples and the values appeared correct?

To clarify when you observe the timestamp of "1969-12-31 23:59:59", is this from the returned JSON string or are you observing this timestamp afterwards in your Python3 code?

[andybrnr]

Hi Eric,

Sorry, cut and pasted from the wrong block; that was indeed supposed to be a HistoricalDataRequest. The timestamp 1969-12-31 23:59:59 that I get is in the returned JSON string. Server is running on Windows 7 x64; I'll try and get the exact versions of all the dependencies installed tomorrow.

[andybrnr]

Screenshot:

[ericvw]

What is your node --version? I have been using v0.12.7 for my testing. I am wondering if there are different NodeJS environments at play here.

[andybrnr]

I'm also on v0.12.7. From some googling and stack overflow references, I feel like this might be caused by some JS datetime library buried in the node dependency stack which is expecting seconds instead of milliseconds or a similar issue, but I'm at a bit of a loss here.

[ericvw]

@andybrnr, what OS are you running on?

[andybrnr]

win7x64

[ericvw]

The root cause is due to how mktime() behaves on Windows. The documentation for WIN32 mktime indicates that for struct tm* before January 1, 1970, that -1 is returned. Which, when -1 is passed into V8::Date::New() it results in one second behind the Unix epoch (i.e., '1969-12-31 23:59:59').

I need to dig a bit deeper to figure out how we can resolve this in blpapi-node, which blpapi-http depends upon, for Windows. blpapi-node uses the this call here.

[andybrnr]

Hmm... chalk up another "win" for MS. I did some looking at other classes that might be applicable. Does something like this look workable https://msdn.microsoft.com/en-us/library/38wh24td.aspx ? Irritating to rely on a specific compilation under MSVS, but since it's windows support anyway, maybe it's not the end of the world?

[andybrnr]

Hi Eric,

I managed to patch this issue by plugging in the library here: http://www.codeproject.com/Articles/4674/Time-bit-Times-for-bit-Windows-Apps

Only change is adding includes for time64.h and time64.c below line 30 in blpapijs.cpp and changing the mktime() call on line 1088 to mktime64(). I made the changes to blpapijs.cpp, used npm rebuild, and then did a fresh make build of blpapi-http, and validated by grabbing SPX since 1928.

Libs seem to be legit for use as no explicit license is specified, but not sure what appropriate protocol is here. I can push a changeset to blpapi-node if you like.

[ericvw]

Unfortunately libraries are not legitimate to use if there is no explicit license specified. If no license is specified, default copyright laws apply. In some cases, this can be even more dangerous because the original author retains all rights to the source code and nobody else may reproduce, distribute, or create derivative works of the work.

Don't worry, this issue is still on my radar.