fix(rtfcre): escape characters outside code page 1252 on export - #1857
Merged
Conversation
Saving a dictionary as RTF encodes the file as code page 1252, so any translation containing a character outside that code page raised UnicodeEncodeError and the export failed. The RTF spec encodes such characters with the \uN control word, using UTF-16 code units, so characters outside the BMP become a surrogate pair. Escape them on save, wrapped in a group setting \uc0, and handle \uN (and \ucN) when parsing so RTF dictionaries still round-trip. Tests cover formatting, saving and loading of BMP characters, surrogate pairs, and the negative \uN values the spec allows.
mkrnr
reviewed
Jul 26, 2026
\ucN sets how many fallback characters follow each \uN escape, and 1 is the RTF default, so files written by other programs emit `\u7903 ?` and the `?` was ending up in the translation. Track the count per group, restore it on group end, and drop that many following characters (control-word substitutions and plain text alike).
mkrnr
approved these changes
Aug 4, 2026
mkrnr
left a comment
mkrnr
left a comment
Contributor
There was a problem hiding this comment.
Thank you for this contribution and for implementing the suggestion!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Saving a dictionary as RTF encodes the file as code page 1252, so any translation containing a character outside that code page (e.g.
ởinphở, or a combining acute accent) raisedUnicodeEncodeErrorand the export failed.Characters that cp1252 cannot represent are now escaped with the RTF
\uNcontrol word using UTF-16 code units, so characters outside the BMP are written as a surrogate pair — the approach @sammdot described on the issue (surrogate pairs, positive numbers, wrapped in\uc0so no ANSI fallback is expected). The parser learned\uN/\ucNto match, so RTF dictionaries still round-trip.Closes #1705
Pull Request Checklist