Skip to content

[feat](iceberg) support aliyun dlf iceberg rest catalog#60796

Open
morningman wants to merge 15 commits intoapache:masterfrom
morningman:wt-dlf-iceberg
Open

[feat](iceberg) support aliyun dlf iceberg rest catalog#60796
morningman wants to merge 15 commits intoapache:masterfrom
morningman:wt-dlf-iceberg

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 22, 2026

What problem does this PR solve?

depends on #60856

Support Aliyun DLF (Data Lake Formation) as an Iceberg REST catalog.

Main changes:

  • Preserve case-sensitive REST signing names by removing toLowerCase()on rest.signing-name, so names like "DlfNext" work correctly.
  • Prioritize non-S3 storage properties when multiple S3-compatible
    properties are returned from iceberg rest catalog, to avoid
    incorrectly using S3 credentials for non-S3 storage (e.g. OSS).
  • Remove redundant resetToUninitialized() override in HMSExternalCatalog
    that only delegates to super.
  • Add IcebergDlfRestCatalogTest, IcebergRestPropertiesTest and
    StoragePropertiesTest for DLF REST catalog integration and property
    resolution testing.
CREATE CATALOG ice PROPERTIES (
    'type' = 'iceberg',
    'warehouse' = 'new_dlf_iceberg_catalog',
    'iceberg.catalog.type' = 'rest',
    'iceberg.rest.uri' = 'http://cn-beijing-vpc.dlf.aliyuncs.com/iceberg',
    'iceberg.rest.sigv4-enabled' = 'true',
    'iceberg.rest.signing-name' = 'DlfNext',
    'iceberg.rest.access-key-id' = 'xx',
    'iceberg.rest.secret-access-key' = 'xx',
    'iceberg.rest.signing-region' = 'cn-beijing',
    'iceberg.rest.vended-credentials-enabled' = 'true',
    'io-impl' = 'org.apache.iceberg.rest.DlfFileIO',
    'fs.oss.support' = 'true'
);

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 22, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28652 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52a5758a05b23c91c136d111088ab2a46293f185, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17608	4500	4289	4289
q2	q3	10644	757	512	512
q4	4685	359	248	248
q5	7563	1207	1034	1034
q6	173	177	148	148
q7	781	844	677	677
q8	9293	1469	1339	1339
q9	4923	4756	4710	4710
q10	6794	1857	1617	1617
q11	467	251	235	235
q12	709	561	458	458
q13	17784	4203	3387	3387
q14	231	228	205	205
q15	953	806	782	782
q16	742	711	668	668
q17	715	890	430	430
q18	5901	5324	5269	5269
q19	1095	959	630	630
q20	495	488	384	384
q21	4362	1893	1385	1385
q22	341	288	245	245
Total cold run time: 96259 ms
Total hot run time: 28652 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4444	4329	4347	4329
q2	q3	1760	2160	1708	1708
q4	830	1145	749	749
q5	3978	4287	4313	4287
q6	176	174	138	138
q7	1718	1587	1466	1466
q8	2389	2651	2507	2507
q9	7183	8007	7537	7537
q10	2760	2937	2451	2451
q11	518	453	417	417
q12	485	586	440	440
q13	3924	4435	3585	3585
q14	279	294	277	277
q15	860	816	809	809
q16	710	772	694	694
q17	1199	1552	1292	1292
q18	7060	6867	6630	6630
q19	954	879	888	879
q20	2104	2164	2012	2012
q21	4036	3500	3382	3382
q22	494	483	411	411
Total cold run time: 47861 ms
Total hot run time: 46000 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183465 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52a5758a05b23c91c136d111088ab2a46293f185, data reload: false

query5	5122	641	504	504
query6	332	223	221	221
query7	4226	467	287	287
query8	374	252	233	233
query9	8770	2731	2721	2721
query10	549	401	330	330
query11	17018	17482	17267	17267
query12	197	133	124	124
query13	1282	493	357	357
query14	7201	3316	3058	3058
query14_1	2865	2851	3203	2851
query15	198	194	175	175
query16	971	448	451	448
query17	1117	772	608	608
query18	3325	454	369	369
query19	227	215	183	183
query20	152	130	152	130
query21	220	140	121	121
query22	5414	5367	4959	4959
query23	17170	16848	16582	16582
query23_1	16721	16623	16716	16623
query24	7140	1624	1232	1232
query24_1	1236	1221	1220	1220
query25	554	490	431	431
query26	1222	261	158	158
query27	2779	462	284	284
query28	4493	1880	1838	1838
query29	793	556	460	460
query30	321	241	199	199
query31	869	720	637	637
query32	78	68	67	67
query33	514	327	275	275
query34	915	935	556	556
query35	640	710	599	599
query36	1076	1105	938	938
query37	124	89	84	84
query38	2950	2864	2872	2864
query39	876	877	836	836
query39_1	832	831	821	821
query40	229	148	135	135
query41	62	58	58	58
query42	107	102	100	100
query43	367	372	357	357
query44	
query45	199	185	183	183
query46	876	988	615	615
query47	2118	2127	2038	2038
query48	304	303	229	229
query49	624	462	374	374
query50	669	272	217	217
query51	4078	4066	3993	3993
query52	108	107	94	94
query53	288	332	280	280
query54	294	263	253	253
query55	89	84	84	84
query56	314	302	315	302
query57	1378	1325	1304	1304
query58	282	266	304	266
query59	2589	2637	2550	2550
query60	326	337	316	316
query61	147	145	147	145
query62	619	592	545	545
query63	306	283	269	269
query64	4847	1278	996	996
query65	
query66	1382	448	341	341
query67	16384	16619	16305	16305
query68	
query69	396	302	281	281
query70	1023	977	942	942
query71	332	307	295	295
query72	2849	2638	2437	2437
query73	529	549	322	322
query74	9945	9928	9733	9733
query75	2854	2745	2433	2433
query76	2288	1044	689	689
query77	370	384	313	313
query78	11146	11288	10729	10729
query79	2730	798	635	635
query80	1840	634	560	560
query81	556	275	253	253
query82	1022	148	117	117
query83	345	264	245	245
query84	265	126	108	108
query85	965	577	436	436
query86	426	300	303	300
query87	3097	3108	2986	2986
query88	3516	2661	2646	2646
query89	424	367	343	343
query90	1952	170	164	164
query91	160	154	133	133
query92	86	72	69	69
query93	1171	842	511	511
query94	640	301	305	301
query95	571	386	311	311
query96	630	529	228	228
query97	2500	2461	2413	2413
query98	232	226	226	226
query99	1020	988	914	914
Total cold run time: 257338 ms
Total hot run time: 183465 ms

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 90.00% (9/10) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 50.00% (5/10) 🎉
Increment coverage report
Complete coverage report

@morningman morningman force-pushed the wt-dlf-iceberg branch 3 times, most recently from 0087590 to a8411a5 Compare February 26, 2026 10:53
Comment on lines 191 to 197

protected void tryAddingIOImpl(Map<String, String> options) {
if (!Strings.isNullOrEmpty(ioImpl)) {
options.put(CatalogProperties.FILE_IO_IMPL, ioImpl);
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be built centrally in the base class, rather than making every subclass implement it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@morningman morningman marked this pull request as ready for review February 27, 2026 03:52
CalvinKirs
CalvinKirs previously approved these changes Feb 27, 2026
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 28, 2026
@doris-robot
Copy link

TPC-H: Total hot run time: 28792 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9bcee2808c2120d23386a86dc98c63ec045d6974, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17654	4575	4308	4308
q2	q3	10652	774	519	519
q4	4675	356	253	253
q5	7549	1186	1025	1025
q6	171	172	144	144
q7	778	854	662	662
q8	9290	1445	1312	1312
q9	4830	4700	4673	4673
q10	6773	1856	1650	1650
q11	449	268	235	235
q12	679	578	467	467
q13	17760	4202	3419	3419
q14	237	233	209	209
q15	932	783	783	783
q16	764	707	677	677
q17	736	875	406	406
q18	5974	5267	5318	5267
q19	1108	979	613	613
q20	516	487	390	390
q21	4675	1965	1469	1469
q22	373	346	311	311
Total cold run time: 96575 ms
Total hot run time: 28792 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4702	4535	4607	4535
q2	q3	1830	2218	1759	1759
q4	879	1179	762	762
q5	4011	4400	4393	4393
q6	186	186	143	143
q7	1824	1673	1514	1514
q8	2508	2669	2536	2536
q9	7574	7349	7373	7349
q10	2621	2850	2406	2406
q11	497	470	509	470
q12	506	597	439	439
q13	3935	4425	3717	3717
q14	288	314	291	291
q15	878	780	806	780
q16	753	747	743	743
q17	1194	1535	1281	1281
q18	7102	6942	6776	6776
q19	872	872	861	861
q20	2071	2209	2025	2025
q21	4013	3474	3355	3355
q22	457	427	375	375
Total cold run time: 48701 ms
Total hot run time: 46510 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184148 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9bcee2808c2120d23386a86dc98c63ec045d6974, data reload: false

query5	5091	644	565	565
query6	337	220	206	206
query7	4226	470	272	272
query8	347	246	240	240
query9	8768	2693	2729	2693
query10	556	409	357	357
query11	17008	17447	17233	17233
query12	199	134	124	124
query13	1262	475	367	367
query14	7097	3379	3222	3222
query14_1	2941	2975	3024	2975
query15	209	207	188	188
query16	1014	466	482	466
query17	1086	802	672	672
query18	2950	469	376	376
query19	230	216	205	205
query20	144	143	133	133
query21	224	142	134	134
query22	5338	4956	4805	4805
query23	17236	16772	16632	16632
query23_1	16805	16784	16771	16771
query24	7185	1614	1220	1220
query24_1	1234	1245	1229	1229
query25	571	490	432	432
query26	1234	281	156	156
query27	2725	472	280	280
query28	4449	1841	1826	1826
query29	803	567	467	467
query30	303	244	201	201
query31	881	732	643	643
query32	80	71	71	71
query33	504	337	276	276
query34	916	912	569	569
query35	647	673	598	598
query36	1050	1150	914	914
query37	125	98	85	85
query38	2954	2909	2940	2909
query39	892	862	848	848
query39_1	821	829	821	821
query40	231	150	135	135
query41	62	61	59	59
query42	106	101	103	101
query43	388	372	341	341
query44	
query45	204	187	186	186
query46	882	974	594	594
query47	2133	2172	2048	2048
query48	313	309	227	227
query49	633	460	377	377
query50	683	288	212	212
query51	4053	4139	4051	4051
query52	108	108	97	97
query53	290	339	281	281
query54	293	262	265	262
query55	87	84	82	82
query56	313	355	304	304
query57	1352	1332	1266	1266
query58	286	277	273	273
query59	2531	2637	2551	2551
query60	341	332	319	319
query61	157	141	177	141
query62	612	593	519	519
query63	318	273	271	271
query64	4901	1251	980	980
query65	
query66	1383	442	348	348
query67	16306	16255	16234	16234
query68	
query69	393	304	289	289
query70	983	963	934	934
query71	333	296	283	283
query72	2825	2641	2337	2337
query73	532	544	317	317
query74	9968	10043	9812	9812
query75	3110	2730	2447	2447
query76	2306	1019	664	664
query77	352	391	303	303
query78	11227	11463	10732	10732
query79	1615	756	603	603
query80	1391	599	569	569
query81	572	281	250	250
query82	985	150	115	115
query83	346	257	241	241
query84	250	115	98	98
query85	891	504	436	436
query86	426	326	320	320
query87	3169	3083	3018	3018
query88	3553	2643	2622	2622
query89	434	380	350	350
query90	1959	171	170	170
query91	166	152	132	132
query92	79	79	73	73
query93	989	840	494	494
query94	660	315	302	302
query95	582	341	372	341
query96	638	510	225	225
query97	2446	2516	2377	2377
query98	224	220	226	220
query99	1004	986	924	924
Total cold run time: 255982 ms
Total hot run time: 184148 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants