TST: windows failures after #7907 (failing 27-32,26-32,26-64)

[jreback]

(ignore the pandas 0.12.0 install, this is on master, its in a separate build dir)

cc @sinhrks
related #7907

C:\Users\Jeff Reback\Documents\GitHub\pandas>  more versions.27-32.log

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.5.final.0
python-bits: 32
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.12.0
nose: 1.3.0
Cython: 0.19.2
numpy: 1.9.0b1
scipy: None
statsmodels: None
IPython: 1.1.0
sphinx: None
patsy: 0.2.1
scikits.timeseries: 0.91.3
dateutil: 2.2
pytz: 2013.8
bottleneck: 0.7.0
tables: None
numexpr: None
matplotlib: 1.3.1
openpyxl: 1.7.0
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: 0.5.0
lxml: 3.3.5
bs4: 4.3.2
html5lib: 1.0b3
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.3
pymysql: 0.6.1.None
psycopg2: 2.5.2 (dt dec pq3 ext)

C:\Users\Jeff Reback\Documents\GitHub\pandas>more test.27-32.log
..SS..SS..SS..SS..SS..SS..SSSSSSSSSS..SS..SS..SSSS...........SS..SS..SS..SS..SS..SS..SS................SSSSSSSSSSSSSSSS..SS...........................
.S...S.S....SSS..............................................S......................................................S.................................
....................S..................................................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS........................................
......................................................................................................................................................
......................................................................................................................................................
.........S...........................................................................................................................................S
............S.....................................................................................................................SS......SSSSSSSSSSSS
SSSSSSSSSSSSSSSSSSSSSSSSSS.....S........................................................................S.............................SSSSSSSSSSS.....
......................................................................................................................................................
.........................S..........S...................................S..................S..S.SSSSSSS.......................................S.......
...S......S.....SSSSSSSSSSSSSSSSSSSSSSSSSSS...........................................................................................................
.........................................................S............................................................................................
................................................S..S..................................................................................................
............................................................................S...S.....................................................................
......................................................................................................................................................
....................................................................................................................F.................................
......................................................................S...............................................................................
..................................................................................S...................................................................
................FFF...F...................................................................................c:\Users\Jeff Reback\Documents\GitHub\pandas
\build\lib.win32-2.7\pandas\core\algorithms.py:140: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
  sorter = uniques.argsort()
C:\python27-32\lib\site-packages\numpy\core\fromnumeric.py:791: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
  a.sort(axis, kind, order)
................................................................................................................S.....................................
......................................................................................................................C:\python27-32\lib\site-packages
\numpy\core\_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
....................................................................................................................................S.SSSSS...........
......................................................................................................................................................
........S.........................................S.S.................................................................................................
...................................S...........................................................................................SSS...SSSS.S.S.S.S.S.S.
SSS.SSS.S.S.S.S.S.S.S...S.S.SSSSSSSSS..............S.....S..SS..S......S.......S..............SS.SSSSS.S....S................c:\Users\Jeff Reback\Docu
ments\GitHub\pandas\build\lib.win32-2.7\pandas\tools\plotting.py:3004: UserWarning: To output multiple subplots, the figure containing the passed axes
 is being cleared
  "is being cleared", UserWarning)
......................................................................................................................................................
................................................................................................................c:\Users\Jeff Reback\Documents\GitHub\
pandas\build\lib.win32-2.7\pandas\core\index.py:1200: RuntimeWarning: Cannot compare type 'Timestamp' with type 'str', sort order is undefined for inc
omparable objects
  "incomparable objects" % e, RuntimeWarning)
......................................................c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\core\index.py:1200: Runt
imeWarning: Cannot compare type 'Timestamp' with type 'long', sort order is undefined for incomparable objects
  "incomparable objects" % e, RuntimeWarning)
......................................................................................................................................................
..............................................S.....S................................................................S................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
......................................................................................................................................................
............................................................................S.........................................................................
...S.S....S........SS.......................................................................................................S.........................
....S...S.............SSS.........S...........SS.............SSS....SSSS..............................................................................
..........................................SS..............................................................................S.......................S...
...........................SS............................................S............................................................................
..........................................................................................S.......
======================================================================
FAIL: test_constructor_datetime64_tzformat (pandas.tseries.tests.test_timeseries.TestDatetimeIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\tseries\tests\test_timeseries.py", line 2182, in test_constructor_date
time64_tzformat
    tm.assert_index_equal(idx, expected)
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\util\testing.py", line 579, in assert_index_equal
    right.dtype))
AssertionError: [index] left [datetime64[ns] <class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 00:00:00-05:00, ..., 2016-01-01 00:00:00-05:00]
Length: 4, Freq: AS-JAN, Timezone: tzoffset(None, -18000)], right [<class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 00:00:00-05:00, ..., 2016-01-01 00:00:00-05:00]
Length: 4, Freq: AS-JAN, Timezone: pytz.FixedOffset(-300) datetime64[ns]]

======================================================================
FAIL: test_constructor (pandas.tseries.tests.test_tslib.TestTimestamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\tseries\tests\test_tslib.py", line 47, in test_constructor
    self.assertEqual(result.value, expected)
AssertionError: 1404205200000000000L != 1404205200000000005L

======================================================================
FAIL: test_constructor_with_stringoffset (pandas.tseries.tests.test_tslib.TestTimestamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\tseries\tests\test_tslib.py", line 100, in test_constructor_with_strin
goffset
    self.assertEqual(result.value, expected)
AssertionError: 1404205200000000000L != 1404205200000000005L

======================================================================
FAIL: test_repr (pandas.tseries.tests.test_tslib.TestTimestamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\tseries\tests\test_tslib.py", line 166, in test_repr
    self.assertIn(tz_repr, repr(date_tz))
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\util\testing.py", line 143, in assertIn
    assert a in b, "%s: %r is not in %r" % (msg.format(a,b), a, b)
AssertionError: : '/US/Pacific' is not in "Timestamp('2014-03-07 00:00:00-0800', tz='dateutil/America/Los_Angeles')"

======================================================================
FAIL: test_nanosecond_string_parsing (pandas.tseries.tests.test_tslib.TestTimestampNsOperations)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-2.7\pandas\tseries\tests\test_tslib.py", line 461, in test_nanosecond_string_pars
ing
    self.assertEqual(ts.value, expected_value)
AssertionError: 1367392545123456000L != 1367392545123456789L

[sinhrks]

Hmm, output looks to be parsed by dateutil, not by cython function. Will check on my windows machine later.

[jreback]

ok great thanks.

[sinhrks]

Could reproduce failure in test_repr using Windows 7 / Python 2.7.6, other tests are passed.

test_repr problem looks unrelated to #7907, but revealed because of the test case. I'll check whether this is caused either by pandas or dateutil. Is it better to fix the test case for now?

>>> import pandas as pd
>>> import dateutil

# parse nanosecond is OK
>>> pd.Timestamp('2011-01-01 09:00:00.000000001')
Timestamp('2011-01-01 09:00:00.000000001')

# Using pytz is OK
>>> pd.Timestamp('2011-01-01 09:00:00.000000001', tz='US/Eastern')
Timestamp('2011-01-01 09:00:00.000000001-0500', tz='US/Eastern')

# parse nanosecond is OK, but dateutil results in different timezone
>>> pd.Timestamp('2011-01-01 09:00:00.000000001', tz='dateutil/US/Pacific')
Timestamp('2011-01-01 09:00:00.000000001-0800', tz='dateutil/America/Los_Angeles')

[jreback]

yes, make this a separate fix, hmm other tests passed? are we using same versions of datetueil/pytz?

[sinhrks]

Hmm... looks like a dateutil issue but can be fixed in pandas side.

If it is only the problem of display name, modifying the following condition seems to solve the case. But I'm not confident with it. Easy fix is to change the test case to use dateutil/America/Los_Angels.
https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L1139

@dbew Any ideas?

@jreback Regarding nanoseconds problem, is it possible that the existing cython file remains (or compile failure)? Because np_datetime_string.c is changed, all the cache should be once flushed.

In Windows 7 / Python 2.7.6 / dateutil 2.2

import pandas as pd
from dateutil import zoneinfo

zoneinfo.gettz('US/Pacific'))
# tzfile('America/Los_Angeles')

# dateutil result is incorrect, and also maybe_get_tz 
pd.tslib.maybe_get_tz('dateutil/US/Pacific')
# tzfile('America/Los_Angeles')

Mac OS 10.9.4 / Python 2.7.6 / dateutil 2.2

zoneinfo.gettz('US/Pacific'))
# tzfile('America/Los_Angeles')

# dateutil result is incorrect, but maybe_get_tz is OK
pd.tslib.maybe_get_tz('dateutil/US/Pacific')
# tzfile('/usr/share/zoneinfo/US/Pacific')

[jreback]

@sinhrks let me make clean and see

btw....do we need to update setup.py to clean that c file somehow?

[sinhrks]

Changing setup.py always forces re-compile and increase an installation time, correct?

How about update timestamps of algos.pyx, index.pyx and lib.pyx to force cython to re-compile once.

[jreback]

the only issue I guess if you have an existing np_datetime.c and rebase to master, then you need to clean first, so no big deal I guess. I guess the *.pyx didn't change but the *.c did so it doesn't pick this up. no big deal, I think it will be ok.

on windows, now only test_repr is failing, so that was the problem.!

[dbew]

@sinhrks That looks weird. There were quite a few differences between windows and linux but the mostly came down to where dateutil looked for the timezone info (because it uses system data). I don't recall anything like the example you've shown there.

Part of the difference might be that maybe_get_tz is calling dateutil.tz.gettz while your example uses dateutil.zoneinfo.gettz

I'm not sure about modifying the line in tslib, it's a while since I looked - what did you change it to? For background, I added that to work around problems when dateutil loaded tz files from an archive but didn't put the right zone info on. Without the zone name, we can't serialize the timezone object.

[jreback]

@sinhrks did we have a fix for this?

[sinhrks]

@dbew

There were quite a few differences between windows and linux but the mostly came down to where dateutil looked for the timezone info (because it uses system data).

Looked dateutil source little, and it uses internal zoneinfo files in Windows cases. Maybe this causes mismatch. Even though the display name is changed, internal object looks different.

In Windows 7 / Python 2.7.6 / dateutil 2.2

>>> t1 = dateutil.tz.gettz('US/Pacific')
>>> t2 = dateutil.tz.gettz('America/Los_Angeles')
>>> t1
tzfile('America/Los_Angeles')
>>> t2
tzfile('America/Los_Angeles')
>>> t1 is t2
False

>>> t1_2 = dateutil.tz.gettz('US/Pacific')
>>> t1_2
tzfile('America/Los_Angeles')

>>> t1 is t1_2
True

Thus one possibility is to change the following line to:
https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L1139

            if isinstance(tz, _dateutil_tzfile) and tz._filename != zone:
                tz._filename = zone

But if there is any concern, leave it as dateutil issue and only fix the test case to use non-problematic zone name.

[dbew]

@sinhrks Your description of the dateutil behaviour matches my memory of it. I wasn't very clear in my earlier comment, sorry.

In linux, tz._filename will be the actual filename e.g. /usr/zoneinfo/US/Pacific not just US/Pacific so I think your test will always evaluate to True. I'm not sure what the effect of changing the filename be in those cases - possibly just a bit confusing that there's a difference between maybe_get_tz and dateutil.tz.gettz. AFAIK, it's not used for timezone conversions and the only place we use it in pandas is for serializing timezone info (and in the long run, I think it might be better to use pickle.dumps for that instead of just the filename).

You could try it and see..