Fix logic for quoting special characters

[perlpunk]

Fixes #275

Currently some strings with special characters are dumped as plain strings, on systems where sys.maxunicode <= 0xffff (typically Python 2.7 on Win/Mac) and when using allow_unicode=True. That doesn't roundtrip.

>>> import yaml
>>> string = "\tpart1\tpart2"
>>> print(yaml.dump(string, allow_unicode=True))

        part1   part2
...

[perlpunk]

While this fixes issue #275, I'm still not sure if the logic is correct now for systems where sys.maxunicode < 0xffff.
Will add an example later.

[perlpunk]

# coding=utf-8
import yaml

string1 = "part1\tpart2".decode('utf8')
string2 = "ü".decode('utf8')
string3 = "part1\rpart2"
string4 = "😀".decode('utf8')
data = [string1, string2, string3, string4]
print(data)
res = yaml.safe_dump(data,
                     default_flow_style=False, explicit_start=True, canonical=False,
                     allow_unicode=True, encoding='utf-8', width=float("inf"))
res = res.decode('utf8')
print(res)
newdata = yaml.safe_load(res)
print(newdata)

Tested on system with sys.maxunicode > 0xffff

Output with my fix no. 1:

[u'part1\tpart2', u'\xfc', 'part1\rpart2', u'\U0001f600']
---
- "part1\tpart2"
- ü
- "part1\rpart2"
- "\U0001F600"

['part1\tpart2', u'\xfc', 'part1\rpart2', u'\U0001f600']

Output if I leave out has_ucs4 from the condition:

[u'part1\tpart2', u'\xfc', 'part1\rpart2', u'\U0001f600']
---
- "part1\tpart2"
- ü
- "part1\rpart2"
- 😀

['part1\tpart2', u'\xfc', 'part1\rpart2', u'\U0001f600']

I think we don't need has_ucs4 here, but I'm not a unicode expert, especially not in python. But if sys.maxunicode <= 0xffff then (u'\U00010000' <= ch < u'\U0010ffff') can't actually be true. So I'm not sure what the has_ucs4 was supposed to do in this if condition.

I also tried it out on a Mac with python 2.7.

[perlpunk]

Pushed my second fix

[perlpunk]

@peterkmurphy Do you remember what the purpose of has_ucs4 was in your original PR?

or ((not has_ucs4) or (u'\U00010000' <= ch < u'\U0010ffff'))) and ch != u'\uFEFF':

[peterkmurphy]

Dear Tina, If I recall correctly, it's related to how Unicode characters are represented in Python. https://stackoverflow.com/questions/1446347/how-to-find-out-if-python-is-compiled-with-ucs-2-or-ucs-4 Originally. PyYaml could only support characters up to 65535; my patch extended it so it go go beyond, but only if the version of Python running would support it. Best regards, Peter

…

On Sat, Mar 23, 2019 at 12:47 AM Tina Müller (tinita) < ***@***.***> wrote: @peterkmurphy <https://github.com/peterkmurphy> Do you remember what the purpose of has_ucs4 was in your original PR? or ((not has_ucs4) or (u'\U00010000' <= ch < u'\U0010ffff'))) and ch != u'\uFEFF': — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#276 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-AtRFpo8dKnah9dNEU7YOcJgKQv34Aks5vZOzqgaJpZM4b4iFk> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[perlpunk]

Thanks @peterkmurphy
The purpose of the PR is clear; my question was more what the has_ucs4 was supposed to do in the if statement.
The problem is, on those systems where it is true, basically the whole condition became true, for characters like tabs, for example. As a result, a string like "\tstring" was dumped as a plain string which didn't roundtrip. See #275
I fixed it by taking the has_ucs4 out and can't think of a case where it would be needed.
Here's the current statement from emitter.py:

    if (ch == u'\x85' or u'\xA0' <= ch <= u'\uD7FF'
            or u'\uE000' <= ch <= u'\uFFFD'
            or ((not has_ucs4) or (u'\U00010000' <= ch < u'\U0010ffff'))) and ch != u'\uFEFF':
        unicode_characters = True
        if not self.allow_unicode:
            special_characters = True
    else:
        special_characters = True

[peterkmurphy]

Dear Tina, If all tests work without using has_ucs4, then it sounds like removing it is the best thing to do. In that case, I think "has_ucs4" can be removed from the if statements. Best regards, Peter

…

On Sat, Mar 23, 2019 at 8:48 PM Tina Müller (tinita) < ***@***.***> wrote: Thanks @peterkmurphy <https://github.com/peterkmurphy> The purpose of the PR is clear; my question was more what the has_ucs4 was supposed to do in the if statement. The problem is, on those systems where it is true, basically the whole condition became true, for characters like tabs, for example. As a result, a string like "\tstring" was dumped as a plain string which didn't roundtrip. See #275 <#275> I fixed it by taking the has_ucs4 out and can't think of a case where it would be needed. Here's the current statement from emitter.py: if (ch == u'\x85' or u'\xA0' <= ch <= u'\uD7FF' or u'\uE000' <= ch <= u'\uFFFD' or ((not has_ucs4) or (u'\U00010000' <= ch < u'\U0010ffff'))) and ch != u'\uFEFF': unicode_characters = True if not self.allow_unicode: special_characters = True else: special_characters = True — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#276 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-AtbACjZgvmEEf14IDB6Ju0e0KLbfkks5vZgaUgaJpZM4b4iFk> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[perlpunk]

ok thanks @peterkmurphy