ASCII code in text > 128 decimal causes crash

[pkunz]

If a value in a field contains an ASCII code > 128 decimal, Python crashes.

[chinyeungli]

For instance,
Component name: GoogleæGuiceæExtensionsæServlet

will cause the encoding error.

The tool will crash with nothing log in the error.log.

[pkunz]

This page might be helful: https://docs.python.org/2/howto/unicode.html

[chinyeungli]

Note that the gen command works fine. i,e, if the input CSV contains value of GoogleæGuiceæExtensionsæServlet, the tool can handle it correctly. However, if an ABOUT file contains GoogleæGuiceæExtensionsæServlet , the tool will crash.

[pombredanne]

Reading an about file with PyYAML should work though. The expectation is this would be UTF-8 but we should be able to recover from weirdly encoded files too, or at least provide proper diagnostic errors.

[chinyeungli]

It's weird that we have code to catch this error

input_text = codecs.open(loc, encoding='utf-8').read()
    errs = self.load_dict(saneyaml.load(input_text), base_dir)
    ...
except Exception, e:
    msg = 'Cannot load invalid ABOUT file: %(location)r: %(e)r'

The code catch the Google® alright. However, it doesn't cause exception for GoogleæGuiceæExtensionsæServlet and therefore crash the tool with

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 113: ordinal not in range(128)

when writing to the output.