Support UTF32 character set

0 Note

0.1 编码

• UTF8, GBK等都是变长编码, 一个字符占1~4字节.
  • 在单字节ASCII范围(0~127)内, 上述编码兼容.
• UTF32为定长4-Byte编码.

0.2 MySQL参数和含义

SELECT CHARACTER_SET_NAME, COLLATION_NAME, ID FROM INFORMATION_SCHEMA.COLLATIONS ORDER BY ID;
show variables like 'character%';
show variables like 'collation%';

https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html :

• character_set_client: The character set for statements that arrive from the client.
• character_set_connection: The character set used for literals specified without a character set introducer and for
  number-to-string conversion.
  • introducer: _utf8'abc' COLLATE utf8_danish_ci
• character_set_database: set only by server
• character_set_filesystem: This variable is used to interpret string literals that refer to file names.
  • LOAD DATA and SELECT ... INTO OUTFILE statements and etc
  • Such file names are converted from character_set_client to character_set_filesystem
• character_set_results: The character set used for returning query results to the client.
• character_set_server: The servers default character set.
• character_set_system: The value is always utf8.

其中 character_set_system为global only. 其余为global/session.

0.3 dtle 行为

• Extractor.readMySqlCharsetSystemVariables() 获取character|collation_set_server, 并生成set statement.
• Applier.ApplyEventQueries()在所有连接上执行(a.db和a.dbs[i])

Problem 1

parser增加utf32支持后, 初步复制效果如下

mysql> select id, val, length(val) from a.a;
+----+------------------------------+-------------+
| id | val                          | length(val) |
+----+------------------------------+-------------+
|  1 | English                      |          28 |
|  2 | 中文                         |           8 |
|  3 | ガンダム                     |          16 |
// 以上全量, 以下增量
|  4 |    E   n   g   l   i   s   h |         112 |
|  5 | 中文                         |           8 |
|  6 | ガンダム                     |          16 |

dtle 中有 ConvertArg() 在增量时进行encoding转换. 增加utf32后正常. 注: MySQL使用BigEndian.

TODO

• 测试 CJK table.column 名 -- 另见 non-ASCII / Chinese table name #733
• 测试client端各种字符集

~ $ mysql -h 10.186.62.40 -P3307 -uroot -ppassword --default-character-set=utf32
ERROR 1231 (42000): Variable 'character_set_client' can't be set to the value of 'utf32'

@@character_set_client只能设为ASCII兼容编码 link

MySQL以character_set_connection解读字符串字面量. 例如用go mysql-client这样插入数据

	encodedString, _, err := transform.String(simplifiedchinese.GBK.NewEncoder(), "中文")
	_, err = conn.ExecContext(ctx, "insert into a.a values (0, ?)", encodedString)

character_set_connection不为gbk则会乱码.