python 2 bytes/str issue

[satra]

in CommandLine interface it terminal_output is allatonce, we do the following:

https://github.com/nipy/nipype/blob/master/nipype/interfaces/base.py#L1257

however on python 2, when the underlying command generates an error the returned string (at least on centos 7), contains some unsavory bytes. this results in an error as seen below

one solution is to detect whether we are on python 2 and not decode the string. any ideas how to solve this for both platforms simultaneously? @matthew-brett @bcipolli

In [45]: from subprocess import Popen, PIPE

In [46]: proc = Popen('cp foo goo', shell=True, stdout=PIPE, stderr=PIPE)

In [47]: stdout, stderr = proc.communicate()

In [48]: stderr
Out[48]: 'cp: cannot stat \xe2\x80\x98foo\xe2\x80\x99: No such file or directory\n'

In [49]: str(stderr)
Out[49]: 'cp: cannot stat \xe2\x80\x98foo\xe2\x80\x99: No such file or directory\n'

In [50]: stderr.decode()
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-50-80fc7c733b30> in <module>()
----> 1 stderr.decode()

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 16: ordinal not in range(128)

In [51]: stderr.decode('ISO-8859-1')
Out[51]: u'cp: cannot stat \xe2\x80\x98foo\xe2\x80\x99: No such file or directory\n'

In [53]: str(stderr.decode('ISO-8859-1'))
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-53-e11a9dfd7944> in <module>()
----> 1 str(stderr.decode('ISO-8859-1'))

UnicodeEncodeError: 'ascii' codec can't encode characters in position 16-18: ordinal not in range(128)

[matthew-brett]

That's weird - I don't get that on OSX. How about using a list instead of string for the arguments - do you get the same error?

[matthew-brett]

Testing on my Centos 7 system, this is because of the fancy open / close quotes around the filename in the error (same when running the command from bash). I guess the system is returning the string using its locale settings:

>>> locale.getdefaultlocale()[1]
'UTF-8'

So I guess you should do something like:

stderr.decode(locale.getdefaultlocale()[1])

on Pythons 2 or 3, to get the right error message.

[satra]

thanks @matthew-brett - indeed it's because of those quotes. but the problem is that in python 2, after i decode, it becomes unicode, which i can't turn into a str (see the last error above). it's possible that the extra conversion to str is unnecessary. will check it out.

also if you have any suggestions how to write a test for this scenario that would be helpful.

[matthew-brett]

Why would you need to convert to str in Python 2?

[satra]

we wouldn't need to but we were trying to come up a way to do the right thing in both py2 and 3. i think i found what was incorrect last time and at least the basic tests have passed. will now wait for circle ci to finish.

[matthew-brett]

Then I guess the right thing to do is to convert to unicode in both Python 2 and Python 3, by decoding using the current locale. You'll have a unicode object in Python 2 and a str in Python 3 (which is also unicode).

[satra]

i think that's what i am now doing in #1394