Fix mojibake when double byte text is written under a single byte font encoding

[EugineD]

Problem

When an encapsulated-HTML RTF body declares a double byte document code page (\ansicpg932, \ansicpg949, ...) but part of the text is written while a \fcharset0 font is active, that text is silently turned into mojibake. There is no ? and no U+FFFD, because a single byte code page maps nearly every byte, so the corruption is easy to miss.

This is #529. Using the test1.msg attached there, the body text あああ (U+3042 x3, Shift-JIS bytes 82A0 82A0 82A0) comes out of BodyHtml as U+201A U+00A0 x3 — which is exactly those same bytes decoded as cp1252.

The reporter noticed it happens after a NO-BREAK SPACE, and that turns out to be mechanically correct. Word emits the   as {\f3\'a0}, and \f3 is the only \fcharset0 font involved; the Japanese text on the next line carries no font of its own and inherits that single byte encoding.

Cause

TryDecode is only ever called from one place:

if (FontTable.MixedEncodings && _runtimeEncoding.IsSingleByte && byteBuffer.Count > 1 && byteBuffer[0] >= 0x80)
    stringBuilder.Append(TryDecode(byteBuffer));

So by construction, whenever it runs, a high byte has arrived while a single byte font encoding is active. The runtime encoding is already suspect — that is the whole reason TryDecode exists — so falling back to it cannot recover the text.

Charset detection does not help here either: the byte buffer is flushed per high byte run, so CharsetDetector typically sees only about two bytes and detects nothing. Execution drops to the fallback. (Lowering CharsetDetectionEncodingConfidenceLevel therefore has no effect — I tested 0.90 down to 0.01 and the output was byte-identical.)

Fix

Prefer the declared document code page in the one case where it is provably better — the runtime encoding is single byte and cannot represent double byte characters, while the document code page is multi byte and can:

if (!_documentCodePageDeclared || _defaultEncoding == null ||
    _defaultEncoding.IsSingleByte || !RuntimeEncoding.IsSingleByte)
    return RuntimeEncoding;

return _defaultEncoding;

This is intentionally narrow, so existing behaviour is preserved everywhere else:

• both encodings single byte (e.g. \ansicpg1252 with a \fcharset204 font) — unchanged;
• runtime encoding multi byte — unchanged;
• no \ansicpg declared — unchanged, via the new _documentCodePageDeclared flag. This guard matters because _defaultEncoding is seeded with Encoding.Default, which is UTF-8 (multi byte) on .NET Core, and would otherwise start winning for documents that never declared a code page.

Successful detection paths are untouched — a confident detection found in the font table still wins.

Tests

Three tests added to RtfDocumentTests, all synthetic inline RTF, no binary fixtures:

Test	Purpose
MixedCharsetFontsWithNoBreakSpaceBeforeDoubleByteText	Reproduces #529 — \ansicpg932 with the  -induced {\f1\'a0} font switch
MixedCharsetFontsWithDoubleByteDocumentCodePage	The same defect with \ansicpg949 (Korean)
MixedCharsetFontsWithSingleByteDocumentCodePageKeepsFontEncoding	Guard — a Cyrillic \fcharset204 font under \ansicpg1252 must keep the font encoding

Both new failing tests genuinely fail on unpatched master (Expected string length 3 but was 6 and 5 but was 10 — each double byte character becoming two single byte ones), and the guard test passes both before and after, confirming the change is narrow.

Full suite passes on net462, net8.0, net9.0 and net10.0: 59 passed, 0 failed, 2 skipped (up from 56). MsgReader.csproj builds clean in Release across all six target frameworks.

Side observation (not fixed here)

{\f3\'a0} is a scoped group, so the font should revert at the }. It does not: GroupStart and GroupEnd are no-ops in the main loop, and _runtimeEncoding is never saved or restored with group nesting, so the font's encoding leaks past the group. That is a separate and older issue, which was harmless while the fallback was the document code page. I have left it alone to keep this PR focused, but it may be worth a look.