jsony doesn't handle UTF-16 surrogate pairs

As of now, jsony doesn't recognize surrogate pairs and just keeps them as-is, resulting in invalid UTF-8 characters:

import std/json
import pkg/jsony

type
  TestObj = object
    content: string

let
  # A string with 🔒 emoji encoded both as normal UTF-8 and as a surrogate pair
  raw = """{"content":"\uD83D\uDD12🔒"}"""
  parsed = raw.fromJson(TestObj)
  parsedStd = parseJson(raw).to(TestObj)

echo "jsony - ", parsed.content
echo "std/json - ", parsedStd.content

Both std/json and nim-json-serialization handle surrogates, so I think jsony should also support them.

Found out about this bug from a discussion with @zedeus.

[zedeus]

Here's a longer test case with more than one surrogate pair:

let raw = """{"content":"\u00A1\uD835\uDC7D\uD835\uDC96\uD835\uDC86\uD835\uDC8D\uD835\uDC97\uD835\uDC86\uD835\uDC8F \uD835\uDC8F\uD835\uDC96\uD835\uDC86\uD835\uDC94\uD835\uDC95\uD835\uDC93\uD835\uDC90\uD835\uDC94 \uD835\uDC89\uD835\uDC8A\uD835\uDC8F\uD835\uDC84\uD835\uDC89\uD835\uDC82\uD835\uDC94!"}"""

jsony - ¡������������������������������������������ ������������������������������������������������ ������������������������������������������!
std/json - ¡𝑽𝒖𝒆𝒍𝒗𝒆𝒏 𝒏𝒖𝒆𝒔𝒕𝒓𝒐𝒔 𝒉𝒊𝒏𝒄𝒉𝒂𝒔!

[treeform]

That feels like a bug. Why don't I run into it? Almost all json is utf8 encoded? Its rare to get UTF-16 surrogate pairs?

[zedeus]

Twitter does it unfortunately, I need this to use jsony for Nitter.

[treeform]

I have fixed the surrogate pairs. Thank you for the bug report.

If it still a problem feel free to reopen with new test case.