MySQL 5.6/8.0 charset

[denji]

Synchronized with the MySQL version 8.0 #492

MySQL 8.0 will have much more complete unicode support based on Unicode 9.0

• xxx_unicode_520_ci collations are based on UCA 5.2.0 weight keys. Collations without the "520", are based on the older UCA 4.0.0.
• MySQL 5.7 already provide a much newer utf8mb4_unicode520_ci collation that handles characters. But in MySQL 8.0 we decided to take this a step further by introducing utf8mb4_0900_ai_ci, which is based on the latest Unicode standard, and we intend to set this collation as the default collation for utf8mb4.
• This is part of a bigger effort to improve our UTF8 support MySQL 8.0. The overall goal is to change the default character set from latin1 to utf8mb4, and the default collation from latin1_swedish_ci to utf8mb4_0900_ai_ci.

• utf8_unicode_ci & utf8mb4_unicode_ci (2003-11) is based on UCA 4.0.0 weight keys.
• utf8mb4_unicode_520_ci (2009-09) is based on UCA 5.2.0 weight keys.
• utf8mb4_unicode_800_ci (2015-02) is based on UCA 8.0.0 weight keys.
• utf8mb4_unicode_0900_ci (2016-05) is based on UCA 9.0.0 weight keys.

mysql> STATUS;
--------------
mysql  Ver 14.14 Distrib 8.0.0-dmr, for Linux (x86_64) using  EditLine wrapper

mysql> SHOW COLLATION;
+----------------------------+----------+-----+---------+----------+---------+
| Collation                  | Charset  | Id  | Default | Compiled | Sortlen |
+----------------------------+----------+-----+---------+----------+---------+
| armscii8_bin               | armscii8 |  64 |         | Yes      |       1 |
| armscii8_general_ci        | armscii8 |  32 | Yes     | Yes      |       1 |
| ascii_bin                  | ascii    |  65 |         | Yes      |       1 |
| ascii_general_ci           | ascii    |  11 | Yes     | Yes      |       1 |
| big5_bin                   | big5     |  84 |         | Yes      |       1 |
| big5_chinese_ci            | big5     |   1 | Yes     | Yes      |       1 |
| binary                     | binary   |  63 | Yes     | Yes      |       1 |
| cp1250_bin                 | cp1250   |  66 |         | Yes      |       1 |
| cp1250_croatian_ci         | cp1250   |  44 |         | Yes      |       1 |
| cp1250_czech_cs            | cp1250   |  34 |         | Yes      |       2 |
| cp1250_general_ci          | cp1250   |  26 | Yes     | Yes      |       1 |
| cp1250_polish_ci           | cp1250   |  99 |         | Yes      |       1 |
| cp1251_bin                 | cp1251   |  50 |         | Yes      |       1 |
| cp1251_bulgarian_ci        | cp1251   |  14 |         | Yes      |       1 |
| cp1251_general_ci          | cp1251   |  51 | Yes     | Yes      |       1 |
| cp1251_general_cs          | cp1251   |  52 |         | Yes      |       1 |
| cp1251_ukrainian_ci        | cp1251   |  23 |         | Yes      |       1 |
| cp1256_bin                 | cp1256   |  67 |         | Yes      |       1 |
| cp1256_general_ci          | cp1256   |  57 | Yes     | Yes      |       1 |
| cp1257_bin                 | cp1257   |  58 |         | Yes      |       1 |
| cp1257_general_ci          | cp1257   |  59 | Yes     | Yes      |       1 |
| cp1257_lithuanian_ci       | cp1257   |  29 |         | Yes      |       1 |
| cp850_bin                  | cp850    |  80 |         | Yes      |       1 |
| cp850_general_ci           | cp850    |   4 | Yes     | Yes      |       1 |
| cp852_bin                  | cp852    |  81 |         | Yes      |       1 |
| cp852_general_ci           | cp852    |  40 | Yes     | Yes      |       1 |
| cp866_bin                  | cp866    |  68 |         | Yes      |       1 |
| cp866_general_ci           | cp866    |  36 | Yes     | Yes      |       1 |
| cp932_bin                  | cp932    |  96 |         | Yes      |       1 |
| cp932_japanese_ci          | cp932    |  95 | Yes     | Yes      |       1 |
| dec8_bin                   | dec8     |  69 |         | Yes      |       1 |
| dec8_swedish_ci            | dec8     |   3 | Yes     | Yes      |       1 |
| eucjpms_bin                | eucjpms  |  98 |         | Yes      |       1 |
| eucjpms_japanese_ci        | eucjpms  |  97 | Yes     | Yes      |       1 |
| euckr_bin                  | euckr    |  85 |         | Yes      |       1 |
| euckr_korean_ci            | euckr    |  19 | Yes     | Yes      |       1 |
| gb18030_bin                | gb18030  | 249 |         | Yes      |       1 |
| gb18030_chinese_ci         | gb18030  | 248 | Yes     | Yes      |       2 |
| gb18030_unicode_520_ci     | gb18030  | 250 |         | Yes      |       8 |
| gb2312_bin                 | gb2312   |  86 |         | Yes      |       1 |
| gb2312_chinese_ci          | gb2312   |  24 | Yes     | Yes      |       1 |
| gbk_bin                    | gbk      |  87 |         | Yes      |       1 |
| gbk_chinese_ci             | gbk      |  28 | Yes     | Yes      |       1 |
| geostd8_bin                | geostd8  |  93 |         | Yes      |       1 |
| geostd8_general_ci         | geostd8  |  92 | Yes     | Yes      |       1 |
| greek_bin                  | greek    |  70 |         | Yes      |       1 |
| greek_general_ci           | greek    |  25 | Yes     | Yes      |       1 |
| hebrew_bin                 | hebrew   |  71 |         | Yes      |       1 |
| hebrew_general_ci          | hebrew   |  16 | Yes     | Yes      |       1 |
| hp8_bin                    | hp8      |  72 |         | Yes      |       1 |
| hp8_english_ci             | hp8      |   6 | Yes     | Yes      |       1 |
| keybcs2_bin                | keybcs2  |  73 |         | Yes      |       1 |
| keybcs2_general_ci         | keybcs2  |  37 | Yes     | Yes      |       1 |
| koi8r_bin                  | koi8r    |  74 |         | Yes      |       1 |
| koi8r_general_ci           | koi8r    |   7 | Yes     | Yes      |       1 |
| koi8u_bin                  | koi8u    |  75 |         | Yes      |       1 |
| koi8u_general_ci           | koi8u    |  22 | Yes     | Yes      |       1 |
| latin1_bin                 | latin1   |  47 |         | Yes      |       1 |
| latin1_danish_ci           | latin1   |  15 |         | Yes      |       1 |
| latin1_general_ci          | latin1   |  48 |         | Yes      |       1 |
| latin1_general_cs          | latin1   |  49 |         | Yes      |       1 |
| latin1_german1_ci          | latin1   |   5 |         | Yes      |       1 |
| latin1_german2_ci          | latin1   |  31 |         | Yes      |       2 |
| latin1_spanish_ci          | latin1   |  94 |         | Yes      |       1 |
| latin1_swedish_ci          | latin1   |   8 | Yes     | Yes      |       1 |
| latin2_bin                 | latin2   |  77 |         | Yes      |       1 |
| latin2_croatian_ci         | latin2   |  27 |         | Yes      |       1 |
| latin2_czech_cs            | latin2   |   2 |         | Yes      |       4 |
| latin2_general_ci          | latin2   |   9 | Yes     | Yes      |       1 |
| latin2_hungarian_ci        | latin2   |  21 |         | Yes      |       1 |
| latin5_bin                 | latin5   |  78 |         | Yes      |       1 |
| latin5_turkish_ci          | latin5   |  30 | Yes     | Yes      |       1 |
| latin7_bin                 | latin7   |  79 |         | Yes      |       1 |
| latin7_estonian_cs         | latin7   |  20 |         | Yes      |       1 |
| latin7_general_ci          | latin7   |  41 | Yes     | Yes      |       1 |
| latin7_general_cs          | latin7   |  42 |         | Yes      |       1 |
| macce_bin                  | macce    |  43 |         | Yes      |       1 |
| macce_general_ci           | macce    |  38 | Yes     | Yes      |       1 |
| macroman_bin               | macroman |  53 |         | Yes      |       1 |
| macroman_general_ci        | macroman |  39 | Yes     | Yes      |       1 |
| sjis_bin                   | sjis     |  88 |         | Yes      |       1 |
| sjis_japanese_ci           | sjis     |  13 | Yes     | Yes      |       1 |
| swe7_bin                   | swe7     |  82 |         | Yes      |       1 |
| swe7_swedish_ci            | swe7     |  10 | Yes     | Yes      |       1 |
| tis620_bin                 | tis620   |  89 |         | Yes      |       1 |
| tis620_thai_ci             | tis620   |  18 | Yes     | Yes      |       4 |
| ucs2_bin                   | ucs2     |  90 |         | Yes      |       1 |
| ucs2_croatian_ci           | ucs2     | 149 |         | Yes      |       8 |
| ucs2_czech_ci              | ucs2     | 138 |         | Yes      |       8 |
| ucs2_danish_ci             | ucs2     | 139 |         | Yes      |       8 |
| ucs2_esperanto_ci          | ucs2     | 145 |         | Yes      |       8 |
| ucs2_estonian_ci           | ucs2     | 134 |         | Yes      |       8 |
| ucs2_general_ci            | ucs2     |  35 | Yes     | Yes      |       1 |
| ucs2_general_mysql500_ci   | ucs2     | 159 |         | Yes      |       1 |
| ucs2_german2_ci            | ucs2     | 148 |         | Yes      |       8 |
| ucs2_hungarian_ci          | ucs2     | 146 |         | Yes      |       8 |
| ucs2_icelandic_ci          | ucs2     | 129 |         | Yes      |       8 |
| ucs2_latvian_ci            | ucs2     | 130 |         | Yes      |       8 |
| ucs2_lithuanian_ci         | ucs2     | 140 |         | Yes      |       8 |
| ucs2_persian_ci            | ucs2     | 144 |         | Yes      |       8 |
| ucs2_polish_ci             | ucs2     | 133 |         | Yes      |       8 |
| ucs2_romanian_ci           | ucs2     | 131 |         | Yes      |       8 |
| ucs2_roman_ci              | ucs2     | 143 |         | Yes      |       8 |
| ucs2_sinhala_ci            | ucs2     | 147 |         | Yes      |       8 |
| ucs2_slovak_ci             | ucs2     | 141 |         | Yes      |       8 |
| ucs2_slovenian_ci          | ucs2     | 132 |         | Yes      |       8 |
| ucs2_spanish2_ci           | ucs2     | 142 |         | Yes      |       8 |
| ucs2_spanish_ci            | ucs2     | 135 |         | Yes      |       8 |
| ucs2_swedish_ci            | ucs2     | 136 |         | Yes      |       8 |
| ucs2_turkish_ci            | ucs2     | 137 |         | Yes      |       8 |
| ucs2_unicode_520_ci        | ucs2     | 150 |         | Yes      |       8 |
| ucs2_unicode_ci            | ucs2     | 128 |         | Yes      |       8 |
| ucs2_vietnamese_ci         | ucs2     | 151 |         | Yes      |       8 |
| ujis_bin                   | ujis     |  91 |         | Yes      |       1 |
| ujis_japanese_ci           | ujis     |  12 | Yes     | Yes      |       1 |
| utf16le_bin                | utf16le  |  62 |         | Yes      |       1 |
| utf16le_general_ci         | utf16le  |  56 | Yes     | Yes      |       1 |
| utf16_bin                  | utf16    |  55 |         | Yes      |       1 |
| utf16_croatian_ci          | utf16    | 122 |         | Yes      |       8 |
| utf16_czech_ci             | utf16    | 111 |         | Yes      |       8 |
| utf16_danish_ci            | utf16    | 112 |         | Yes      |       8 |
| utf16_esperanto_ci         | utf16    | 118 |         | Yes      |       8 |
| utf16_estonian_ci          | utf16    | 107 |         | Yes      |       8 |
| utf16_general_ci           | utf16    |  54 | Yes     | Yes      |       1 |
| utf16_german2_ci           | utf16    | 121 |         | Yes      |       8 |
| utf16_hungarian_ci         | utf16    | 119 |         | Yes      |       8 |
| utf16_icelandic_ci         | utf16    | 102 |         | Yes      |       8 |
| utf16_latvian_ci           | utf16    | 103 |         | Yes      |       8 |
| utf16_lithuanian_ci        | utf16    | 113 |         | Yes      |       8 |
| utf16_persian_ci           | utf16    | 117 |         | Yes      |       8 |
| utf16_polish_ci            | utf16    | 106 |         | Yes      |       8 |
| utf16_romanian_ci          | utf16    | 104 |         | Yes      |       8 |
| utf16_roman_ci             | utf16    | 116 |         | Yes      |       8 |
| utf16_sinhala_ci           | utf16    | 120 |         | Yes      |       8 |
| utf16_slovak_ci            | utf16    | 114 |         | Yes      |       8 |
| utf16_slovenian_ci         | utf16    | 105 |         | Yes      |       8 |
| utf16_spanish2_ci          | utf16    | 115 |         | Yes      |       8 |
| utf16_spanish_ci           | utf16    | 108 |         | Yes      |       8 |
| utf16_swedish_ci           | utf16    | 109 |         | Yes      |       8 |
| utf16_turkish_ci           | utf16    | 110 |         | Yes      |       8 |
| utf16_unicode_520_ci       | utf16    | 123 |         | Yes      |       8 |
| utf16_unicode_ci           | utf16    | 101 |         | Yes      |       8 |
| utf16_vietnamese_ci        | utf16    | 124 |         | Yes      |       8 |
| utf32_bin                  | utf32    |  61 |         | Yes      |       1 |
| utf32_croatian_ci          | utf32    | 181 |         | Yes      |       8 |
| utf32_czech_ci             | utf32    | 170 |         | Yes      |       8 |
| utf32_danish_ci            | utf32    | 171 |         | Yes      |       8 |
| utf32_esperanto_ci         | utf32    | 177 |         | Yes      |       8 |
| utf32_estonian_ci          | utf32    | 166 |         | Yes      |       8 |
| utf32_general_ci           | utf32    |  60 | Yes     | Yes      |       1 |
| utf32_german2_ci           | utf32    | 180 |         | Yes      |       8 |
| utf32_hungarian_ci         | utf32    | 178 |         | Yes      |       8 |
| utf32_icelandic_ci         | utf32    | 161 |         | Yes      |       8 |
| utf32_latvian_ci           | utf32    | 162 |         | Yes      |       8 |
| utf32_lithuanian_ci        | utf32    | 172 |         | Yes      |       8 |
| utf32_persian_ci           | utf32    | 176 |         | Yes      |       8 |
| utf32_polish_ci            | utf32    | 165 |         | Yes      |       8 |
| utf32_romanian_ci          | utf32    | 163 |         | Yes      |       8 |
| utf32_roman_ci             | utf32    | 175 |         | Yes      |       8 |
| utf32_sinhala_ci           | utf32    | 179 |         | Yes      |       8 |
| utf32_slovak_ci            | utf32    | 173 |         | Yes      |       8 |
| utf32_slovenian_ci         | utf32    | 164 |         | Yes      |       8 |
| utf32_spanish2_ci          | utf32    | 174 |         | Yes      |       8 |
| utf32_spanish_ci           | utf32    | 167 |         | Yes      |       8 |
| utf32_swedish_ci           | utf32    | 168 |         | Yes      |       8 |
| utf32_turkish_ci           | utf32    | 169 |         | Yes      |       8 |
| utf32_unicode_520_ci       | utf32    | 182 |         | Yes      |       8 |
| utf32_unicode_ci           | utf32    | 160 |         | Yes      |       8 |
| utf32_vietnamese_ci        | utf32    | 183 |         | Yes      |       8 |
| utf8mb4_0900_ai_ci         | utf8mb4  | 255 |         | Yes      |       8 |
| utf8mb4_bin                | utf8mb4  |  46 |         | Yes      |       1 |
| utf8mb4_croatian_ci        | utf8mb4  | 245 |         | Yes      |       8 |
| utf8mb4_cs_0900_ai_ci      | utf8mb4  | 266 |         | Yes      |       8 |
| utf8mb4_czech_ci           | utf8mb4  | 234 |         | Yes      |       8 |
| utf8mb4_danish_ci          | utf8mb4  | 235 |         | Yes      |       8 |
| utf8mb4_da_0900_ai_ci      | utf8mb4  | 267 |         | Yes      |       8 |
| utf8mb4_de_pb_0900_ai_ci   | utf8mb4  | 256 |         | Yes      |       8 |
| utf8mb4_eo_0900_ai_ci      | utf8mb4  | 273 |         | Yes      |       8 |
| utf8mb4_esperanto_ci       | utf8mb4  | 241 |         | Yes      |       8 |
| utf8mb4_estonian_ci        | utf8mb4  | 230 |         | Yes      |       8 |
| utf8mb4_es_0900_ai_ci      | utf8mb4  | 263 |         | Yes      |       8 |
| utf8mb4_es_trad_0900_ai_ci | utf8mb4  | 270 |         | Yes      |       8 |
| utf8mb4_et_0900_ai_ci      | utf8mb4  | 262 |         | Yes      |       8 |
| utf8mb4_general_ci         | utf8mb4  |  45 | Yes     | Yes      |       1 |
| utf8mb4_german2_ci         | utf8mb4  | 244 |         | Yes      |       8 |
| utf8mb4_hr_0900_ai_ci      | utf8mb4  | 275 |         | Yes      |       8 |
| utf8mb4_hungarian_ci       | utf8mb4  | 242 |         | Yes      |       8 |
| utf8mb4_hu_0900_ai_ci      | utf8mb4  | 274 |         | Yes      |       8 |
| utf8mb4_icelandic_ci       | utf8mb4  | 225 |         | Yes      |       8 |
| utf8mb4_is_0900_ai_ci      | utf8mb4  | 257 |         | Yes      |       8 |
| utf8mb4_latvian_ci         | utf8mb4  | 226 |         | Yes      |       8 |
| utf8mb4_la_0900_ai_ci      | utf8mb4  | 271 |         | Yes      |       8 |
| utf8mb4_lithuanian_ci      | utf8mb4  | 236 |         | Yes      |       8 |
| utf8mb4_lt_0900_ai_ci      | utf8mb4  | 268 |         | Yes      |       8 |
| utf8mb4_lv_0900_ai_ci      | utf8mb4  | 258 |         | Yes      |       8 |
| utf8mb4_persian_ci         | utf8mb4  | 240 |         | Yes      |       8 |
| utf8mb4_pl_0900_ai_ci      | utf8mb4  | 261 |         | Yes      |       8 |
| utf8mb4_polish_ci          | utf8mb4  | 229 |         | Yes      |       8 |
| utf8mb4_romanian_ci        | utf8mb4  | 227 |         | Yes      |       8 |
| utf8mb4_roman_ci           | utf8mb4  | 239 |         | Yes      |       8 |
| utf8mb4_ro_0900_ai_ci      | utf8mb4  | 259 |         | Yes      |       8 |
| utf8mb4_sinhala_ci         | utf8mb4  | 243 |         | Yes      |       8 |
| utf8mb4_sk_0900_ai_ci      | utf8mb4  | 269 |         | Yes      |       8 |
| utf8mb4_slovak_ci          | utf8mb4  | 237 |         | Yes      |       8 |
| utf8mb4_slovenian_ci       | utf8mb4  | 228 |         | Yes      |       8 |
| utf8mb4_sl_0900_ai_ci      | utf8mb4  | 260 |         | Yes      |       8 |
| utf8mb4_spanish2_ci        | utf8mb4  | 238 |         | Yes      |       8 |
| utf8mb4_spanish_ci         | utf8mb4  | 231 |         | Yes      |       8 |
| utf8mb4_sv_0900_ai_ci      | utf8mb4  | 264 |         | Yes      |       8 |
| utf8mb4_swedish_ci         | utf8mb4  | 232 |         | Yes      |       8 |
| utf8mb4_tr_0900_ai_ci      | utf8mb4  | 265 |         | Yes      |       8 |
| utf8mb4_turkish_ci         | utf8mb4  | 233 |         | Yes      |       8 |
| utf8mb4_unicode_520_ci     | utf8mb4  | 246 |         | Yes      |       8 |
| utf8mb4_unicode_ci         | utf8mb4  | 224 |         | Yes      |       8 |
| utf8mb4_vietnamese_ci      | utf8mb4  | 247 |         | Yes      |       8 |
| utf8mb4_vi_0900_ai_ci      | utf8mb4  | 277 |         | Yes      |       8 |
| utf8_bin                   | utf8     |  83 |         | Yes      |       1 |
| utf8_croatian_ci           | utf8     | 213 |         | Yes      |       8 |
| utf8_czech_ci              | utf8     | 202 |         | Yes      |       8 |
| utf8_danish_ci             | utf8     | 203 |         | Yes      |       8 |
| utf8_esperanto_ci          | utf8     | 209 |         | Yes      |       8 |
| utf8_estonian_ci           | utf8     | 198 |         | Yes      |       8 |
| utf8_general_ci            | utf8     |  33 | Yes     | Yes      |       1 |
| utf8_general_mysql500_ci   | utf8     | 223 |         | Yes      |       1 |
| utf8_german2_ci            | utf8     | 212 |         | Yes      |       8 |
| utf8_hungarian_ci          | utf8     | 210 |         | Yes      |       8 |
| utf8_icelandic_ci          | utf8     | 193 |         | Yes      |       8 |
| utf8_latvian_ci            | utf8     | 194 |         | Yes      |       8 |
| utf8_lithuanian_ci         | utf8     | 204 |         | Yes      |       8 |
| utf8_persian_ci            | utf8     | 208 |         | Yes      |       8 |
| utf8_polish_ci             | utf8     | 197 |         | Yes      |       8 |
| utf8_romanian_ci           | utf8     | 195 |         | Yes      |       8 |
| utf8_roman_ci              | utf8     | 207 |         | Yes      |       8 |
| utf8_sinhala_ci            | utf8     | 211 |         | Yes      |       8 |
| utf8_slovak_ci             | utf8     | 205 |         | Yes      |       8 |
| utf8_slovenian_ci          | utf8     | 196 |         | Yes      |       8 |
| utf8_spanish2_ci           | utf8     | 206 |         | Yes      |       8 |
| utf8_spanish_ci            | utf8     | 199 |         | Yes      |       8 |
| utf8_swedish_ci            | utf8     | 200 |         | Yes      |       8 |
| utf8_tolower_ci            | utf8     |  76 |         | Yes      |       1 |
| utf8_turkish_ci            | utf8     | 201 |         | Yes      |       8 |
| utf8_unicode_520_ci        | utf8     | 214 |         | Yes      |       8 |
| utf8_unicode_ci            | utf8     | 192 |         | Yes      |       8 |
| utf8_vietnamese_ci         | utf8     | 215 |         | Yes      |       8 |
+----------------------------+----------+-----+---------+----------+---------+

[sidorares]

lgtm

[sidorares]

thanks @denji ! Do you think anything needs to be added to encoding_charset.js ? Or what's there already covers new changes?

[denji]

@sidorares Yes, now everything should be fine. encoding_charset.js based default collation?

[sidorares]

encoding_charset.js based default collation?

it's used to map from numeric code in mysql response to name for iconv-lite. Usually (always?) it's default collation code

[sidorares]

mysql utf8 is in fact 'cesu8' encoding ( while what they call 'utf8mb4' is normal utf8 ), that is why cesu8 was there originally. It would be good to have more tests to validate this, see some discussion here - #374 (comment)