dev/mail#36 - Fix invalid unicode characters in bounce processing

[tschuettler]

Overview

MySQL does not accept invalid unicode characters and therefore fails to write the bounce to the database if it is part of the bounce reason.
Issue tracker: https://lab.civicrm.org/dev/mail/issues/36

Before

Exception is thrown. Bounce is not stored in the datebase, but mail is moved to processed folder.

After

Invalid character are replaced with unicode replacement character. No exception. Bounce is saved.

Technical Details

Generating an invalid unicode character was not easy. Had to use a hex editor and replaced a valid character with an invalid one.

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[tschuettler]

The invalid character is a \x81 in aufgefhrt between f and h.

[tschuettler]

Hmm, it looks like the unit tests inside that file never ran.

@eileenmcnaughton I guess this is due to class and file naming vs. the actual file path?

[eileenmcnaughton]

@tschuettler good spotting - change looks good too

[tschuettler]

@eileenmcnaughton Thanks for merging! :)

What do you think about replacing replacing non-utf8-mysql characters like emojis aswell? They will lead to the same error until we make the switch to utf8mb4.

I looked for other cases where the unit tests name did not match and created an issue for it: https://lab.civicrm.org/dev/core/issues/647

[eileenmcnaughton]

@tschuettler I guess in this case (saving bounce reason) the data is relatively unimportant compared to the fact it's saved - ie. we are not putting this in outgoing messages but a technical field in the db - so if there is a chance other characters could break it it would be good to handle them