[improvement](parser) Add case-insensitive stream fast paths

[morrySnow]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Every SQL lexer currently copies its input through CharStreams.fromString() and calls Character.toUpperCase() for every case-insensitive lookahead. This is visible in lexer allocation and identifier-heavy parsing time.

Add two strictly compatible fast paths:

• Fold ASCII a-z with arithmetic and retain Character.toUpperCase() for every other code point.
• Read strings without UTF-16 surrogates directly, avoiding ANTLR's copied code-point buffer. Inputs containing any surrogate retain the original ANTLR stream so code-point indices, getText, navigation, and errors remain unchanged.

The public arbitrary-CharStream constructor is preserved. All production String entry points now use the factory. ANTLR's native caseInsensitive option was evaluated but rejected because it changed existing Unicode behavior such as the handling of ſelect.

Benchmark

Environment and method:

• Baseline: cf33a08bcd5, artifact SHA-256 51700bef953361218912abc6aec348cecdec3fe61031eba10cc1ebc534bc9183
• Candidate: 7aa7b5e6487, artifact SHA-256 6e8edf78befc0dc349be398aae454c172b5cb968ee98d88a0e056d4c21a9c5d0
• macOS 15.0.1 arm64, OpenJDK 17.0.20.1, 1 GiB heap, JMH 1.37
• 2 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, -prof gc
• Interleaved baseline/candidate runs; one candidate run affected by unrelated host load was discarded and repeated. The table averages two valid runs per artifact. Individual JMH scores use 99.9% confidence intervals.

java -Xms1g -Xmx1g -jar <benchmark.jar> \
  'CaseInsensitiveStreamBenchmark.(createLexer|foldPrebuiltCharacters|parseStatement|tokenize)' \
  -p workload=shortQuery,lowercaseIdentifiers,stringAndComment,unicode \
  -f 2 -wi 4 -i 7 -w 300ms -r 400ms -prof gc -rf json

Benchmark	Baseline us/op (B1 / B2)	Candidate us/op (C1 / C3)	Mean change	Baseline -> candidate B/op
Tokenize 64 lowercase identifiers	10.826±0.078 / 11.408±0.125	8.708±0.071 / 8.704±0.096	-21.7%	14,799 -> 10,464 (-29.3%)
Parse 64 lowercase identifiers	54.775±7.360 / 56.365±4.224	49.860±0.191 / 52.340±1.160	-8.0%	99,395 -> 95,017 (-4.4%)
Tokenize select 1	0.451±0.004 / 0.485±0.011	0.348±0.069 / 0.324±0.010	-28.2%	1,008 -> 816 (-19.0%)
Parse strings/comments with Unicode	27.047±5.371 / 28.435±2.932	24.744±0.401 / 24.743±0.425	-10.8%	24,167 -> 23,784 (-1.6%)
Tokenize supplementary Unicode control	32.034±1.398 / 33.910±0.752	32.303±1.108 / 32.020±0.738	-2.5%	24,630 -> 24,594 (-0.1%)

The isolated createLexer measurement for the surrogate-containing string/comment workload regresses by 5.9% because the compatibility guard scans for surrogates before falling back. The corresponding complete tokenize and parse paths improve by 3.6% and 10.8%; no end-to-end control workload regressed. Prebuilt lowercase character folding improves by 10.6%.

The performance gains come from eliminating the copied input buffer for BMP-only SQL, avoiding its allocation, and replacing the common lowercase ASCII Character.toUpperCase() call with an arithmetic branch.

Correctness corpus:

• 4,610 tracked SQL files; baseline and candidate parse signatures match in legacy and ANSI modes. SHA-256: 69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a.
• 9,220 lexer cases (4,610 SQL files x both noBackslashEscapes modes); complete token tuple and lexer-error signatures match. SHA-256: 66780d4e1d9224ea27c8f715ab33edd247faed1571030ba9e90a8ef3f6212180.

Release note

None

Check List (For Author)

• Test
  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

Tested with:

• mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark -Dmaven.build.cache.enabled=false clean package: 170 tests passed; Checkstyle passed.

• ./run-fe-ut.sh --run org.apache.doris.httpv2.websql.SingleStatementValidatorTest,org.apache.doris.nereids.parser.NereidsParserTest,org.apache.doris.nereids.parser.NereidsParserDigestTest: 91 tests passed.

• Token and parse corpus comparisons described above: no differences.

• FE-only ./build.sh --fe with a fresh output directory: passed and produced Successfully build Doris.

• Full ./run-fe-ut.sh was attempted, but the host's Surefire JVM stopped during the pre-existing fe-connector-trino TrinoPredicateConverterTest dynamic-attach path before reaching fe-core; it was terminated after 90 minutes with no P4-related failure.

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morrySnow]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 16942 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7aa7b5e648734ed846679e93886c2d3fb2187002, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17566	3065	3050	3050
q2	2117	264	232	232
q3	10221	926	512	512
q4	4672	252	203	203
q5	7680	569	390	390
q6	137	119	93	93
q7	536	520	384	384
q8	9245	884	928	884
q9	3410	2397	2384	2384
q10	6557	849	732	732
q11	393	203	185	185
q12	608	263	203	203
q13	18133	1508	1147	1147
q14	160	151	142	142
q15	q16	435	397	368	368
q17	1399	875	808	808
q18	3145	2222	2204	2204
q19	1303	865	790	790
q20	406	294	198	198
q21	5622	1805	1827	1805
q22	338	274	228	228
Total cold run time: 94083 ms
Total hot run time: 16942 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3444	3387	3331	3331
q2	506	396	372	372
q3	2182	2245	2103	2103
q4	1193	1155	897	897
q5	2163	2115	2082	2082
q6	168	121	86	86
q7	1012	903	834	834
q8	1598	1404	1408	1404
q9	3136	3099	3112	3099
q10	1897	1777	1629	1629
q11	353	270	252	252
q12	453	426	349	349
q13	1486	1534	1147	1147
q14	171	173	167	167
q15	q16	389	395	366	366
q17	3495	3368	3226	3226
q18	4779	4392	4699	4392
q19	875	883	889	883
q20	1004	958	828	828
q21	3851	3085	3270	3085
q22	411	350	321	321
Total cold run time: 34566 ms
Total hot run time: 30853 ms

[hello-stephen]

TPC-DS: Total hot run time: 81938 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7aa7b5e648734ed846679e93886c2d3fb2187002, data reload: false

query5	4248	411	346	346
query6	399	137	133	133
query7	4925	405	232	232
query8	287	122	116	116
query9	8702	2907	2897	2897
query10	396	218	173	173
query11	5403	1045	921	921
query12	124	73	70	70
query13	1215	446	353	353
query14	6069	2227	2114	2114
query14_1	2011	1969	1998	1969
query15	177	123	110	110
query16	927	373	341	341
query17	799	470	378	378
query18	2345	328	239	239
query19	171	142	115	115
query20	74	71	71	71
query21	203	107	91	91
query22	5416	5575	5198	5198
query23	6730	6158	5969	5969
query23_1	5959	6303	5920	5920
query24	7278	1115	745	745
query24_1	792	785	813	785
query25	447	307	263	263
query26	1242	249	128	128
query27	2778	465	245	245
query28	4649	1496	1489	1489
query29	929	421	346	346
query30	247	159	129	129
query31	821	399	325	325
query32	143	74	73	73
query33	455	214	165	165
query34	1005	810	490	490
query35	391	411	335	335
query36	583	567	543	543
query37	125	82	70	70
query38	996	839	822	822
query39	516	476	488	476
query39_1	477	432	460	432
query40	203	99	79	79
query41	56	55	52	52
query42	75	72	72	72
query43	233	238	213	213
query44	1027	564	559	559
query45	108	104	102	102
query46	751	851	536	536
query47	749	765	696	696
query48	293	311	222	222
query49	521	238	185	185
query50	770	272	201	201
query51	8226	8105	8126	8105
query52	68	67	57	57
query53	194	198	153	153
query54	240	172	159	159
query55	80	62	60	60
query56	213	210	189	189
query57	674	656	677	656
query58	221	162	163	162
query59	1193	1224	1067	1067
query60	234	173	169	169
query61	109	121	119	119
query62	348	200	186	186
query63	173	148	143	143
query64	2809	678	625	625
query65	1657	1599	1614	1599
query66	1839	275	205	205
query67	10536	9744	9696	9696
query68	3001	1255	721	721
query69	358	224	201	201
query70	662	609	613	609
query71	258	174	169	169
query72	2430	1743	1580	1580
query73	613	638	344	344
query74	2007	1238	1126	1126
query75	1181	1098	949	949
query76	2377	740	569	569
query77	260	252	223	223
query78	3768	3513	3147	3147
query79	2847	768	581	581
query80	1593	351	282	282
query81	494	155	132	132
query82	987	135	101	101
query83	302	212	194	194
query84	305	112	96	96
query85	828	357	291	291
query86	414	181	175	175
query87	1023	981	876	876
query88	2774	2117	2102	2102
query89	307	193	174	174
query90	1913	125	128	125
query91	136	132	103	103
query92	82	71	70	70
query93	1608	1036	730	730
query94	664	274	209	209
query95	518	256	306	256
query96	756	620	285	285
query97	1035	1054	1024	1024
query98	164	133	133	133
query99	437	359	306	306
Total cold run time: 179585 ms
Total hot run time: 81938 ms

[hello-stephen]

ClickBench: Total hot run time: 14.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7aa7b5e648734ed846679e93886c2d3fb2187002, data reload: false

query1	0.00	0.00	0.00
query2	0.07	0.03	0.03
query3	0.24	0.11	0.11
query4	1.60	0.09	0.09
query5	0.17	0.15	0.16
query6	1.25	0.67	0.69
query7	0.03	0.00	0.01
query8	0.05	0.03	0.06
query9	0.29	0.21	0.21
query10	0.37	0.34	0.34
query11	0.17	0.11	0.12
query12	0.14	0.11	0.12
query13	0.31	0.31	0.30
query14	0.45	0.45	0.44
query15	0.37	0.35	0.35
query16	0.22	0.23	0.24
query17	0.69	0.73	0.71
query18	0.18	0.18	0.16
query19	1.18	1.08	1.05
query20	0.01	0.01	0.01
query21	15.48	0.15	0.11
query22	5.07	0.04	0.04
query23	16.20	0.25	0.10
query24	3.03	0.31	0.26
query25	0.13	0.05	0.03
query26	0.77	0.17	0.13
query27	0.04	0.02	0.03
query28	3.74	0.56	0.27
query29	12.42	3.19	2.56
query30	0.27	0.11	0.12
query31	2.76	0.38	0.17
query32	3.51	0.33	0.22
query33	1.53	1.38	1.42
query34	15.39	2.25	1.78
query35	1.78	1.74	1.71
query36	0.46	0.29	0.30
query37	0.07	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.07	0.08
query41	0.09	0.02	0.03
query42	0.03	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 90.78 s
Total hot run time: 14.52 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 6.82% (3/44) 🎉
Increment coverage report
Complete coverage report