Possible utf8 <> utf8mb4 confusion

[AtiX]

Hi,

I was debugging a bug in my application and stumbled across something in mysql2 that looks odd to me and would like to clarify.

The bug:

• Context: I am using typeorm and cannot save an emoji (4 unicode bytes) in a string field. mysql2 version 2.2.5
• The server (MySQL 5.7) has the collation set to utf8mb4_unicode_ci, as does the table/column.
• I supply the connection option charset: "utf8mb4_unicode_ci" to mysql2, but I still get an error when trying to save the string
• Only when I manually execute the query SET NAMES 'utf8mb4'; and SET CHARACTER SET utf8mb4; after connecting, it works.

Now, what I found odd in mysql2 (when the connection is configured and instantiated):

• in charsets.js, the string UTF8MB4_UNICODE_CI is mapped to charset number 224
• However, this number is later then mapped to this.clientEncoding = "utf8" via charset_encoding.js, which looks strange to me, as charset_encoding.js states that is "basically the result of SHOW COLLATION query" (in which case the encoding should be utf8mb4, not utf8)

I see that there is special mapping code in generate-charset-mapping.js to map utf8mb4 onto utf8 so this is probably deliberate(?),

but in the end, there still seems to be a bug, as I cannot use the utf8mb4 charsets without "manual patching" after the connection is already established.

[sidorares]

can you prepare simplest possible example to reproduce the bug?

utf8mb4 vs utf8: utf8mb4 is internal mysql name for what's actually is a utf8 encoding. We need to map mysql names to encoding name to be used together with iconv. See #374 (comment) for some additional information

[dod38fr]

utf8mb4 vs utf8: utf8mb4 is internal mysql name for what's actually is a utf8 encoding.

Err, no.

MySql doc specifies:

• utf8mb4: A UTF-8 encoding of the Unicode character set using one to four bytes per character.
• utf8mb3: A UTF-8 encoding of the Unicode character set using one to three bytes per character.
• utf8: An alias for utf8mb3.

I.e Mysql's utf8 does not support all unicode characters, only those which can be encoded in 3 bytes. An emoji is encoded with 4 bytes and cannot be encoded in Mysql's utf8, but only in Mysql's utf8mb4

HTH