Adding support to Unicode characters over codepoint 0xffff

[peterkmurphy]

This patch is aimed at solving issue #25 . As a side effect, the testing code has been trimmed to accommodate the fixes. Why have I done that? I might as well repeat what I said on gmail:

The problem I am having is with the testing code, which will need a lot of unwrangling. There are some frankly ... "bizarre" assumptions in what won't work in parseable code, except it sometimes will be parseable by accident. For example:

def test_unicode_input_errors(unicode_filename, verbose=False):
    data = open(unicode_filename, 'rb').read().decode('utf-8')
    for input in [data.encode('latin1', 'ignore'), # <--- Look at this!
                    data.encode('utf-16-be'), data.encode('utf-16-le'),
                    codecs.BOM_UTF8+data.encode('utf-16-be'),
                    codecs.BOM_UTF16_BE+data.encode('utf-16-le'),
                    codecs.BOM_UTF16_LE+data.encode('utf-8')+'!']:
        try:
            yaml.load(input)
        except yaml.YAMLError, exc:
            if verbose:
                print exc
        else:
            raise AssertionError("expected an exception")

The idea: let's cause some bizarre combinations of byte sequences, attempt to parse it, and if it doesn't throw a YAMLError, raise an exception. Except that when one does data.encode('latin1', 'ignore') on data, one results in ten line breaks, which is happily parseable as YAML. So no exception raised, so AssertionError.

What should I do in this case - remove test_unicode_input_errors from the PyYaml testing code? Yes, the number of tests will go down, which is generally not a good thing, but if the tests are based on dodgy assumptions...

In some cases I have altered testing code; others I have removed them.

[adamchainz]

removing trailing newlines is bad form as it's diff noise. I'd undo it for all the files you've added it for, which seems to be every file you've touched. Also files should end with a trailing newline to be POSIX valid

[adamchainz]

trimming trailing whitespace is bad form as it's diff noise

[peterkmurphy]

Adam, Please feel free to do the changes as you see fit. I don't know why they happened at all. Perhaps the editor I was using (Atom on Linux Mint x64) is to blame. Best regards, Peter

…

On Wed, May 10, 2017 at 7:21 PM, Adam Johnson ***@***.***> wrote: ***@***.**** requested changes on this pull request. ------------------------------ In README <#63 (comment)>: > @@ -32,4 +32,3 @@ Submit bug reports and feature requests to the PyYAML bug tracker: PyYAML is written by Kirill Simonov ***@***.***>. It is released under the MIT license. See the file LICENSE for more details. - removing trailing newlines is bad form as it's diff noise. I'd undo it for all the files you've added it for, which seems to be every file you've touched. Also files should end with a trailing newline to be POSIX valid ------------------------------ In lib/yaml/emitter.py <#63 (comment)>: > @@ -674,7 +678,7 @@ def analyze_scalar(self, scalar): # Check for indicators. if index == 0: # Leading indicators are special characters. - if ch in u'#,[]{}&*!|>\'\"%@`': + if ch in u'#,[]{}&*!|>\'\"%@`': trimming trailing whitespace is bad form as it's diff noise — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#63 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-AtYWG4MJjDFP9prMvmTILeunl8Nkiks5r4YGDgaJpZM4NWBnY> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[adamchainz]

@peterkmurphy I'm not an admin on this project, I can't do anything to your PR.

[samdmarshall]

you may need to ping @sigmavirus24 or another commiter for this project to get this merged and a new release made.

[adamchainz]

I copied this and tidied it up in #65

[peterkmurphy]

Adam, That's probably for the best. I will keep my pull request up until yours is integrated into the main branch; it contains some explanatory notes. After that, I'll withdraw mine. Thanks for cleaning up. Best regards, Peter

…

On Tue, May 16, 2017 at 7:38 PM, Adam Johnson ***@***.***> wrote: I copied this and tidied it up in #65 <#65> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-Ates03dkvncNnSz2JgYVMklsghcJTks5r6W6jgaJpZM4NWBnY> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[sigmavirus24]

Thanks @peterkmurphy! 🎉 ✨

[jborean93]

@ingydotnet is there any chance this fix could be backported to the 3.x branch and a new release made? I can install the pre-release 4.x builds but I haven't seen any action recently that indicates a full release will be made on those changes.

cc @nitzmahone

[nitzmahone]

I'd be +1 for that

[ingydotnet]

@perlpunk and I are meeting up in a week. We might be able to discuss it then.