Problem with sending emails with emojis

[spoof]

I got an issue with emails that contain emojis. When I receive email with emojis
all emojies are displayed like a bunch of '??'. I tried to send the same data via Postman
and email I received wasn't broken. So it seems someting wrong happens on the way
between this python client and API server.
By digging into source code of Transmissions part and by sniffing traffic
I found that python's json module encodes (dumps) 'not simple' non-ascii characters
(such emojies) to '\uXXXX' notation as a surrogate pair (as expected):

>>> emoji = u"😄"
>>> emoji
u'\U0001f604'
>>> print json.dumps(emoji)
"\ud83d\ude04"

By sending this data I got this emoji looks like:

\xed\xa0\xbd\xed\xb8\x84

And it's of course wrong. I sniffed traffic between Postman and API server and I found
that Postman sends json in utf-8 (without escaping to \uXXXX):

\xf0\x9f\x98\x84

So I repeat this in Python and it worked:

>>> json.dumps(emoji, ensure_ascii=False).encode('utf-8')
'"\xf0\x9f\x98\x84"'

I don't know on what side problem is, but this is my workaround to handle this problem.
And this Issue is just a point on a problem and its possible solution.

PS. There is no problem with sending emails consist of simple chars like '\u2605'.

[richleland]

Thanks for reporting and investigating @spoof. We'll dig in as soon as we can on this.

[richleland]

@spoof I just pushed a potential fix for this to the tweaks branch if you'd like to try it out.

[arturfsousa]

Hey @richleland thanks for the last bumped version, now the bcc and cc are working good. The problem I have now is with the ascii json.dumps. As my language is pt-BR the code is braking when a send a payload with unicode data! Do have any suggestions about it?

I've sent a pull request #129 to enable it back and another fix with substitution_data.

[richleland]

@arturfelipe hrm I may have been too hasty to cut that release then. Can you post a sample of the code you're using so I can test more thoroughly? Also and example of the response or broken email that you're receiving would be great.

[richleland]

Also - that PR looks good - I'm going to merge and cut a new release and then will continue investigating in this issue.

[arturfsousa]

@richleland Thanks man. Here's an example:

sp = SparkPost()
results = sp.transmissions.send(
    from_email='nao-responda@aventurashop.com.br',
    cc=['artur.sousa@corp.globo.com'],
    subject='test with unicode',
    text='test unicode {{ name }}',
    recipients=[{
        'address': {'email': 'arturfelipe.sousa@gmail.com'},
        'substitution_data': {'name': u'João'}
    }])

The unicode name: u'João' with a non ascii char. It raises:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 71: ordinal not in range(128)

I did some testing here encoding the json string generated and it worked for my case and with emojis:

json.dumps(payload, ensure_ascii=False).encode('utf-8')

I can send you a PR with the change if it's ok to you.

[richleland]

Oh that's awesome @arturfelipe please do!

[richleland]

FYI just released 1.3.2: https://github.com/SparkPost/python-sparkpost/releases/tag/v1.3.2

[arturfsousa]

@richleland I'm very appreciated. Just sent a new PR #131 If is there anything i can do, please feel free to let me know.