util/collate: implement utf8mb4_0900_ai_ci collation | tidb-test=pr/2192

[YangKeao]

What problem does this PR solve?

Issue Number: close #37566

Problem Summary:

Add utf8mb4_0900_ai_ci collation to the TiDB.

This PR is 400-600 lines long, quite big. (A lot of lines in the 100k+ are generated or downloaded from unicode.org, so don't be too scared).

What is changed and how it works?

1. Refractor the original unicode_ci implementation to include the code generator in the source files. (Also add tests to make sure they are still the same).
2. Generate data codes for the utf8mb4_0900_ai_ci collation.
3. Re-use most of the implementation from unicode_ci, through code generation.

This PR uses code generation in two places: one for the data (from allkeys.txt), one for the implementation (to reuse the existing unicode_ci collation).

The second choice is hard, a more normal way is to use interface or something to provide different tiny implementation based on a more general UCA framework. However, as I've tested, if the related functions are not inlined (especially when it's an interface method call, or a generic method call), the performance can degrade 20%-50%.

TODO

• Make sure the generated table is the same with MySQL's in uca_data.h.

Benchmark

It becomes faster, and the 0900AICI is faster than UnicodeCI in the benchmark, but I don't know why 🤦 .

Before this PR

BenchmarkUtf8mb4UnicodeCI_CompareShort-8        220942854              319.9 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareMid-8           3252583             22085 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareLong-8             2590          28277470 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyShort-8            362425852              207.8 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyMid-8               5530659             13061 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyLong-8                 4460          15526126 ns/op

After this PR

BenchmarkUtf8mb4UnicodeCI_CompareShort-8        232078950              302.8 ns/op
BenchmarkUtf8mb40900AICI_CompareShort-8         235785336         296.0 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareMid-8           3324604             21180 ns/op
BenchmarkUtf8mb40900AICI_CompareMid-8            3447740             20757 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareLong-8             2659          27342506 ns/op
BenchmarkUtf8mb40900AICI_CompareLong-8              2706          26881637 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyShort-8            368781840              195.1 ns/op
BenchmarkUtf8mb40900AICI_KeyShort-8             400789008              182.1 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyMid-8               5853435             12336 ns/op
BenchmarkUtf8mb40900AICI_KeyMid-8                6267391             11592 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyLong-8                 4704          15383704 ns/op
BenchmarkUtf8mb40900AICI_KeyLong-8                  4426          15551595 ns/op

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Release note

Support the `utf8mb4_0900_ai_ci` collation.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[tiprow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov]

Codecov Report

Merging #45650 (98c7cb3) into master (6ab8a37) will decrease coverage by 1.8383%.
Report is 10 commits behind head on master.
The diff coverage is 90.6474%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #45650        +/-   ##
================================================
- Coverage   74.7681%   72.9298%   -1.8383%     
================================================
  Files          1293       1305        +12     
  Lines        398159     402188      +4029     
================================================
- Hits         297696     293315      -4381     
- Misses        81103      90417      +9314     
+ Partials      19360      18456       -904     

Flag	Coverage Δ
integration	26.0386% <14.3884%> (-20.2244%)	⬇️
unit	73.6056% <90.6474%> (+0.2520%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	54.0444% <ø> (ø)
parser	85.0765% <ø> (+0.0216%)	⬆️
br	47.8367% <ø> (-16.6454%)	⬇️

[YangKeao]

/retest

[YangKeao]

/retest

[YangKeao]

Are we sure the license of allkeys-9.0.0.txt is fine?

@dveeden The license of unicode data file is: https://www.unicode.org/license.txt . It's seems fine.

[dveeden]

Are we sure the license of allkeys-9.0.0.txt is fine?

@dveeden The license of unicode data file is: https://www.unicode.org/license.txt . It's seems fine.

Yes that indeed seems to be fine. I think we should include a copy of the license in the LICENSES/ directory.

$ find LICENSES/
LICENSES/
LICENSES/QL-LICENSE
LICENSES/STRUTIL-LICENSE

Might be good to use the SPDX ID and name this LICENSES/Unicode-DFS-2016-LICENSE.

See also: https://spdx.org/licenses/Unicode-DFS-2016.html

It might be useful to follow what GitHub expects: https://github.blog/changelog/2022-05-26-easily-discover-and-navigate-to-multiple-licenses-in-repositories/

[YangKeao]

select 'foo' collate utf8mb4_0900_ai_ci = 'Foo' collate utf8mb4_general_ci

@dveeden I think this error is compatible with MySQL. MySQL also gives the same error:

mysql> select 'foo' collate utf8mb4_0900_ai_ci = 'Foo' collate utf8mb4_general_ci;
ERROR 1267 (HY000): Illegal mix of collations (utf8mb4_0900_ai_ci,EXPLICIT) and (utf8mb4_general_ci,EXPLICIT) for operation '='

The document also approves this behavior:

MySQL uses coercibility values with the following rules to resolve ambiguities:

• If both sides have the same coercibility, then:
  • If both sides are Unicode, or both sides are not Unicode, it is an error.

[YangKeao]

I've checked that the weight_string is equal between TiDB and MySQL for each characters in utf8mb4_0900_ai_ci, but the default character set of parameters has a little difference. See: #46059

The test program:

package main

import (
	"database/sql"
	"fmt"

	_ "github.com/go-sql-driver/mysql"
	"golang.org/x/exp/slices"
)

func main() {
	dbMySQL, err := sql.Open("mysql", "root:123456@/test")
	if err != nil {
		panic(err)
	}
	dbTiDB, err := sql.Open("mysql", "root:@tcp(127.0.0.1:4000)/test")
	if err != nil {
		panic(err)
	}

	for i := 0; i < 183969; i++ {
		if i%1000 == 0 {
			fmt.Printf("i: %d\n", i)
		}
		rows, err := dbMySQL.Query("SELECT WEIGHT_STRING(? collate utf8mb4_0900_ai_ci)", fmt.Sprintf("%c", rune(i)))
		if err != nil {
			panic(err)
		}
		var expectedWeight []uint8

		for rows.Next() {
			err = rows.Scan(&expectedWeight)
			if err != nil {
				panic(err)
			}
			break
		}
		rows.Close()

		rows, err = dbTiDB.Query("SELECT WEIGHT_STRING(CONVERT(? USING utf8mb4) collate utf8mb4_0900_ai_ci)", fmt.Sprintf("%c", rune(i)))
		if err != nil {
			panic(err)
		}
		var weight []uint8
		for rows.Next() {
			err = rows.Scan(&weight)
			if err != nil {
				panic(err)
			}
			break
		}
		rows.Close()

		if !slices.Equal(expectedWeight, weight) {
			fmt.Printf("The weight of %X is not equal: expectedWeight: %v, weight: %v\n", i, expectedWeight, weight)
		}
	}
}

[YangKeao]

/retest

[CbcWestwolf]

LGTM :-)

[YangKeao]

/retest

[bb7133]

LGTM

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, CbcWestwolf

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CbcWestwolf,bb7133]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-08-14 15:57:50.636533256 +0000 UTC m=+560235.185549240: ☑️ agreed by CbcWestwolf.
• 2023-08-15 05:49:00.370880162 +0000 UTC m=+610104.919896149: ☑️ agreed by bb7133.

[YangKeao]

/retest