[fix](external catalog) Fix missing fields when rebuilding metadata from image #47603

zy-kkk · 2025-02-07T11:43:20Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #41510

Problem Summary:

In PR #41510, we added several fields to External Catalog. However, we only handled the upgrade scenario for EditLog but not for Image. This causes Catalogs rebuilt from Image to miss these fields, resulting in NullPointerException during queries. This PR fixes this issue.

Specifically:

Added null check and initialization for fields in gsonPostProcess
Ensured consistent behavior between EditLog replay and Image deserialization
Added proper logging for better troubleshooting

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-02-07T11:43:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zy-kkk · 2025-02-07T11:49:07Z

run buildall

morningman

LGTM

github-actions · 2025-02-07T12:11:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-07T12:11:39Z

PR approved by anyone and no changes requested.

doris-robot · 2025-02-07T12:22:21Z

TPC-H: Total hot run time: 31533 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6311ff295ff0fbeeff9dd6474e93fd5e8c1d8155, data reload: false

------ Round 1 ----------------------------------
q1	17565	5215	5061	5061
q2	2050	301	172	172
q3	10415	1229	769	769
q4	10280	1026	551	551
q5	8142	2299	2377	2299
q6	188	162	131	131
q7	899	753	595	595
q8	9312	1292	1108	1108
q9	4887	4515	4827	4515
q10	6836	2330	1881	1881
q11	491	285	255	255
q12	350	348	213	213
q13	17752	3667	3119	3119
q14	235	239	208	208
q15	518	462	471	462
q16	621	619	574	574
q17	578	858	343	343
q18	6781	6185	6219	6185
q19	1207	943	569	569
q20	313	317	199	199
q21	2866	2409	2017	2017
q22	368	333	307	307
Total cold run time: 102654 ms
Total hot run time: 31533 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5188	5108	5147	5108
q2	231	331	245	245
q3	2186	2703	2315	2315
q4	1443	1849	1423	1423
q5	4231	4153	4134	4134
q6	212	163	123	123
q7	1859	1793	1709	1709
q8	2576	2699	2634	2634
q9	7195	7130	7159	7130
q10	3001	3213	2825	2825
q11	586	514	479	479
q12	694	812	621	621
q13	3995	3908	3276	3276
q14	285	290	273	273
q15	513	468	457	457
q16	674	660	620	620
q17	1138	1629	1345	1345
q18	7559	7380	7297	7297
q19	793	822	869	822
q20	1980	2031	1896	1896
q21	5418	4929	4733	4733
q22	637	580	559	559
Total cold run time: 52394 ms
Total hot run time: 50024 ms

doris-robot · 2025-02-07T12:34:25Z

TPC-DS: Total hot run time: 190547 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6311ff295ff0fbeeff9dd6474e93fd5e8c1d8155, data reload: false

query1	1317	981	945	945
query2	6233	1880	1854	1854
query3	10983	4461	4437	4437
query4	53674	26020	22984	22984
query5	5044	595	483	483
query6	321	207	189	189
query7	4908	515	296	296
query8	302	248	244	244
query9	5574	2505	2479	2479
query10	438	309	242	242
query11	15097	14989	14879	14879
query12	159	104	103	103
query13	1015	515	365	365
query14	10663	6609	6668	6609
query15	210	185	178	178
query16	7191	639	490	490
query17	1135	734	595	595
query18	1602	461	321	321
query19	209	206	167	167
query20	127	126	127	126
query21	209	129	109	109
query22	4479	4458	4513	4458
query23	34021	33700	33647	33647
query24	5721	2473	2413	2413
query25	484	456	411	411
query26	694	276	162	162
query27	1807	495	357	357
query28	2913	2428	2397	2397
query29	595	563	423	423
query30	226	187	152	152
query31	946	876	837	837
query32	73	62	59	59
query33	445	349	308	308
query34	776	860	519	519
query35	792	860	788	788
query36	938	995	897	897
query37	125	105	86	86
query38	4319	4594	4313	4313
query39	1499	1459	1443	1443
query40	225	116	109	109
query41	49	53	49	49
query42	121	108	108	108
query43	509	524	486	486
query44	1372	789	814	789
query45	180	182	167	167
query46	905	1080	697	697
query47	1838	1863	1779	1779
query48	385	413	302	302
query49	658	529	425	425
query50	718	777	422	422
query51	4233	4311	4194	4194
query52	113	109	102	102
query53	245	270	201	201
query54	478	496	426	426
query55	83	82	81	81
query56	269	288	251	251
query57	1196	1180	1105	1105
query58	252	243	233	233
query59	2738	2979	2908	2908
query60	288	263	261	261
query61	118	116	124	116
query62	772	786	710	710
query63	237	195	192	192
query64	1830	1025	687	687
query65	3273	3134	3156	3134
query66	772	397	293	293
query67	15744	15632	15517	15517
query68	4728	795	526	526
query69	529	349	274	274
query70	1205	1112	1066	1066
query71	414	310	267	267
query72	6304	3627	3933	3627
query73	1163	779	351	351
query74	9049	9057	8819	8819
query75	3214	3165	2696	2696
query76	3805	1172	770	770
query77	558	397	286	286
query78	10006	10171	9330	9330
query79	2909	797	597	597
query80	880	543	469	469
query81	526	275	239	239
query82	525	148	120	120
query83	203	174	151	151
query84	285	102	76	76
query85	794	357	306	306
query86	444	299	292	292
query87	4537	4564	4316	4316
query88	3691	2186	2161	2161
query89	417	328	284	284
query90	1635	196	192	192
query91	136	141	118	118
query92	77	60	61	60
query93	2537	989	586	586
query94	732	417	293	293
query95	345	276	269	269
query96	487	558	270	270
query97	2790	2807	2789	2789
query98	245	197	201	197
query99	1335	1390	1241	1241
Total cold run time: 293370 ms
Total hot run time: 190547 ms

doris-robot · 2025-02-07T12:39:44Z

ClickBench: Total hot run time: 31.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6311ff295ff0fbeeff9dd6474e93fd5e8c1d8155, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.43	0.41	0.41
query6	1.17	0.67	0.65
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.50	0.54
query10	0.58	0.58	0.60
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.62
query14	2.80	2.80	2.74
query15	0.92	0.85	0.86
query16	0.37	0.39	0.38
query17	1.05	1.01	1.03
query18	0.21	0.19	0.19
query19	1.92	1.96	1.75
query20	0.01	0.02	0.01
query21	15.35	0.90	0.54
query22	0.77	1.22	0.69
query23	14.90	1.39	0.62
query24	6.99	1.50	1.51
query25	0.45	0.14	0.13
query26	0.71	0.17	0.15
query27	0.05	0.05	0.05
query28	9.47	0.89	0.42
query29	12.56	4.00	3.34
query30	0.25	0.09	0.06
query31	2.83	0.60	0.38
query32	3.22	0.54	0.46
query33	2.97	3.03	3.01
query34	15.66	5.10	4.53
query35	4.54	4.51	4.51
query36	0.67	0.49	0.47
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.87 s
Total hot run time: 31.3 s

…rom image

morningman · 2025-02-10T09:45:35Z

run buildall

doris-robot · 2025-02-10T10:06:45Z

TPC-H: Total hot run time: 31381 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76d55b2a3bf7b8bd58e9ad49df57e53a5e27942b, data reload: false

------ Round 1 ----------------------------------
q1	17596	5248	5060	5060
q2	2041	319	171	171
q3	10372	1269	704	704
q4	10215	986	514	514
q5	7523	2370	2376	2370
q6	189	168	135	135
q7	891	756	589	589
q8	9300	1299	1087	1087
q9	4874	4636	4581	4581
q10	6825	2317	1870	1870
q11	476	280	249	249
q12	348	351	210	210
q13	17756	3680	3112	3112
q14	224	217	211	211
q15	518	466	462	462
q16	620	632	605	605
q17	575	863	323	323
q18	6717	6224	6217	6217
q19	1749	960	542	542
q20	303	319	188	188
q21	2793	2201	1879	1879
q22	355	324	302	302
Total cold run time: 102260 ms
Total hot run time: 31381 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5178	5148	5076	5076
q2	237	330	238	238
q3	2150	2621	2353	2353
q4	1386	1812	1340	1340
q5	4265	4133	4167	4133
q6	204	162	124	124
q7	1860	1823	1655	1655
q8	2601	2634	2531	2531
q9	7242	7155	7037	7037
q10	2998	3236	2763	2763
q11	588	504	489	489
q12	664	774	610	610
q13	3499	3942	3335	3335
q14	277	300	269	269
q15	522	462	460	460
q16	626	676	624	624
q17	1129	1617	1309	1309
q18	7558	7306	7387	7306
q19	802	831	942	831
q20	1989	1992	1849	1849
q21	5377	5060	4807	4807
q22	630	578	545	545
Total cold run time: 51782 ms
Total hot run time: 49684 ms

doris-robot · 2025-02-10T10:18:11Z

TPC-DS: Total hot run time: 182686 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76d55b2a3bf7b8bd58e9ad49df57e53a5e27942b, data reload: false

query1	957	381	388	381
query2	6545	1907	1876	1876
query3	6797	211	204	204
query4	26525	23152	22994	22994
query5	4326	661	470	470
query6	300	198	198	198
query7	4609	488	306	306
query8	312	242	226	226
query9	8583	2443	2441	2441
query10	460	292	251	251
query11	15819	15199	14832	14832
query12	152	104	106	104
query13	1651	505	388	388
query14	9384	6279	6139	6139
query15	201	195	171	171
query16	7191	628	491	491
query17	1204	699	564	564
query18	1945	406	298	298
query19	197	191	164	164
query20	122	113	114	113
query21	211	124	104	104
query22	4015	4368	4523	4368
query23	34200	33569	33086	33086
query24	7793	2370	2355	2355
query25	583	463	413	413
query26	1238	272	157	157
query27	2429	484	322	322
query28	4188	2381	2347	2347
query29	773	554	439	439
query30	231	186	206	186
query31	941	856	774	774
query32	72	75	59	59
query33	554	350	299	299
query34	785	843	472	472
query35	801	809	730	730
query36	970	981	893	893
query37	118	100	88	88
query38	4048	4173	4174	4173
query39	1434	1466	1440	1440
query40	210	113	102	102
query41	57	52	53	52
query42	117	99	101	99
query43	497	500	495	495
query44	1250	770	768	768
query45	180	169	162	162
query46	867	1050	640	640
query47	1751	1794	1739	1739
query48	372	400	314	314
query49	778	497	399	399
query50	702	728	403	403
query51	4183	4224	4127	4127
query52	107	103	105	103
query53	219	250	183	183
query54	473	483	400	400
query55	82	78	83	78
query56	270	265	240	240
query57	1121	1160	1051	1051
query58	249	227	259	227
query59	2768	2848	2635	2635
query60	278	259	248	248
query61	118	117	116	116
query62	803	727	648	648
query63	219	189	183	183
query64	4370	991	647	647
query65	3231	3132	3134	3132
query66	1125	450	305	305
query67	16011	15548	15300	15300
query68	7669	754	489	489
query69	482	295	262	262
query70	1180	1123	1120	1120
query71	398	294	258	258
query72	6003	3592	3746	3592
query73	747	721	336	336
query74	8862	9113	8674	8674
query75	3161	3149	2672	2672
query76	3272	1157	730	730
query77	476	374	280	280
query78	10038	10050	9165	9165
query79	2804	800	580	580
query80	805	532	453	453
query81	533	269	232	232
query82	744	155	119	119
query83	172	169	154	154
query84	233	92	68	68
query85	810	415	302	302
query86	416	293	282	282
query87	4364	4500	4557	4500
query88	4023	2189	2181	2181
query89	392	313	274	274
query90	1902	194	191	191
query91	205	135	108	108
query92	78	63	56	56
query93	2509	984	558	558
query94	751	401	286	286
query95	344	261	259	259
query96	482	569	266	266
query97	2795	2877	2691	2691
query98	224	216	198	198
query99	1300	1413	1238	1238
Total cold run time: 273246 ms
Total hot run time: 182686 ms

doris-robot · 2025-02-10T10:23:28Z

ClickBench: Total hot run time: 30.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76d55b2a3bf7b8bd58e9ad49df57e53a5e27942b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.41	0.42	0.40
query6	1.21	0.67	0.65
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.61	0.50	0.52
query10	0.57	0.57	0.58
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.62	0.60	0.60
query14	2.83	2.81	2.87
query15	0.90	0.85	0.86
query16	0.39	0.37	0.41
query17	1.05	1.05	1.06
query18	0.21	0.19	0.19
query19	1.94	1.84	1.94
query20	0.01	0.01	0.01
query21	15.38	0.89	0.55
query22	0.75	1.25	0.74
query23	14.78	1.37	0.60
query24	6.99	1.68	0.71
query25	0.51	0.23	0.07
query26	0.69	0.16	0.15
query27	0.06	0.05	0.05
query28	9.19	0.88	0.41
query29	12.61	3.92	3.27
query30	0.25	0.09	0.07
query31	2.81	0.58	0.39
query32	3.22	0.54	0.45
query33	2.96	3.01	2.98
query34	15.69	5.09	4.52
query35	4.51	4.54	4.53
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.6 s
Total hot run time: 30.6 s

morningman · 2025-02-10T10:39:13Z

run buildall

morningman · 2025-02-10T11:08:34Z

run buildall

doris-robot · 2025-02-10T11:27:14Z

TPC-H: Total hot run time: 31595 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44f9706995f699e94785f2bfd1f2446319965dba, data reload: false

------ Round 1 ----------------------------------
q1	17636	5127	5066	5066
q2	2048	311	177	177
q3	10392	1316	698	698
q4	10221	1004	549	549
q5	7522	2401	2328	2328
q6	183	169	135	135
q7	890	738	602	602
q8	9301	1281	1051	1051
q9	4935	4804	4676	4676
q10	6804	2308	1885	1885
q11	462	274	256	256
q12	348	353	219	219
q13	17774	3626	3085	3085
q14	224	233	204	204
q15	515	479	466	466
q16	635	613	585	585
q17	566	880	348	348
q18	7248	6165	6287	6165
q19	1218	961	525	525
q20	313	318	200	200
q21	2735	2127	2059	2059
q22	373	316	319	316
Total cold run time: 102343 ms
Total hot run time: 31595 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5086	5069	5081	5069
q2	237	323	234	234
q3	2204	2681	2259	2259
q4	1407	1823	1381	1381
q5	4187	4145	4120	4120
q6	208	163	124	124
q7	1864	1803	1655	1655
q8	2667	2756	2613	2613
q9	7290	7106	7062	7062
q10	3003	3176	2703	2703
q11	570	496	482	482
q12	676	771	622	622
q13	3554	3852	3267	3267
q14	278	285	276	276
q15	527	473	463	463
q16	645	669	627	627
q17	1133	1573	1336	1336
q18	7518	7426	7326	7326
q19	764	808	877	808
q20	1945	2027	1855	1855
q21	5308	4921	4814	4814
q22	654	567	587	567
Total cold run time: 51725 ms
Total hot run time: 49663 ms

doris-robot · 2025-02-10T11:38:40Z

TPC-DS: Total hot run time: 182592 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44f9706995f699e94785f2bfd1f2446319965dba, data reload: false

query1	959	377	391	377
query2	6542	1915	1851	1851
query3	6788	212	213	212
query4	26223	23317	22932	22932
query5	4329	676	474	474
query6	295	193	189	189
query7	4599	497	303	303
query8	301	244	231	231
query9	8586	2482	2490	2482
query10	476	309	265	265
query11	15401	15166	14750	14750
query12	161	110	117	110
query13	1638	512	373	373
query14	8702	6503	6155	6155
query15	207	192	174	174
query16	7278	654	431	431
query17	1194	686	535	535
query18	1927	397	285	285
query19	188	178	145	145
query20	118	116	113	113
query21	208	128	99	99
query22	4288	4385	4481	4385
query23	34313	33356	33074	33074
query24	7734	2365	2333	2333
query25	534	445	384	384
query26	1235	261	152	152
query27	2392	486	314	314
query28	4176	2392	2352	2352
query29	773	526	406	406
query30	230	181	154	154
query31	930	888	782	782
query32	79	62	58	58
query33	550	342	295	295
query34	772	846	486	486
query35	796	806	729	729
query36	966	951	891	891
query37	122	102	75	75
query38	4169	4150	4009	4009
query39	1430	1382	1396	1382
query40	207	112	103	103
query41	54	52	52	52
query42	119	100	101	100
query43	497	518	470	470
query44	1277	766	773	766
query45	173	170	158	158
query46	840	1039	630	630
query47	1756	1826	1754	1754
query48	372	404	286	286
query49	745	506	424	424
query50	661	746	403	403
query51	4166	4149	4103	4103
query52	109	107	96	96
query53	224	251	179	179
query54	481	474	402	402
query55	80	83	84	83
query56	274	258	253	253
query57	1122	1155	1052	1052
query58	253	241	263	241
query59	2875	2911	2748	2748
query60	319	287	280	280
query61	140	140	145	140
query62	818	710	651	651
query63	231	187	186	186
query64	4531	1109	638	638
query65	3219	3111	3107	3107
query66	1144	400	289	289
query67	15773	15647	15440	15440
query68	8090	753	487	487
query69	486	299	257	257
query70	1228	1040	1080	1040
query71	412	288	272	272
query72	6117	3545	3671	3545
query73	745	703	339	339
query74	8946	9089	9021	9021
query75	3320	3141	2741	2741
query76	3334	1167	744	744
query77	538	417	278	278
query78	9990	9936	9272	9272
query79	2867	788	578	578
query80	620	530	436	436
query81	506	264	241	241
query82	668	149	116	116
query83	174	165	159	159
query84	241	96	72	72
query85	744	338	301	301
query86	373	320	281	281
query87	4389	4510	4268	4268
query88	3706	2168	2162	2162
query89	387	318	279	279
query90	1867	189	202	189
query91	131	137	112	112
query92	78	59	59	59
query93	1991	1013	565	565
query94	715	398	293	293
query95	347	267	258	258
query96	481	567	267	267
query97	2860	2928	2714	2714
query98	221	200	200	200
query99	1444	1387	1265	1265
Total cold run time: 272085 ms
Total hot run time: 182592 ms

doris-robot · 2025-02-10T11:43:58Z

ClickBench: Total hot run time: 31.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44f9706995f699e94785f2bfd1f2446319965dba, data reload: false

query1	0.04	0.04	0.06
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.60	0.10	0.10
query5	0.42	0.43	0.41
query6	1.19	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.54
query10	0.58	0.59	0.57
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.63	0.60	0.61
query14	2.71	2.82	2.82
query15	0.91	0.84	0.83
query16	0.38	0.39	0.38
query17	1.00	1.02	1.02
query18	0.21	0.19	0.20
query19	1.86	1.76	2.05
query20	0.01	0.02	0.01
query21	15.35	0.88	0.55
query22	0.74	1.30	0.91
query23	14.73	1.35	0.60
query24	7.06	1.66	1.63
query25	0.47	0.22	0.06
query26	0.56	0.16	0.15
query27	0.06	0.05	0.05
query28	10.02	0.80	0.42
query29	12.52	3.98	3.28
query30	0.27	0.09	0.06
query31	2.82	0.60	0.38
query32	3.22	0.54	0.48
query33	3.11	3.00	3.04
query34	15.86	5.08	4.50
query35	4.54	4.51	4.55
query36	0.67	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.28 s
Total hot run time: 31.61 s

morningman · 2025-02-10T13:15:50Z

run buildall

github-actions · 2025-02-10T15:43:34Z

PR approved by at least one committer and no changes requested.

suxiaogang223

+1

…g metadata from image (#47726) cherry-pick #47603

…rom image (apache#47603) Issue Number: close #xxx Related PR: apache#41510 Problem Summary: In PR apache#41510, we added several fields to External Catalog. However, we only handled the upgrade scenario for EditLog but not for Image. This causes Catalogs rebuilt from Image to miss these fields, resulting in NullPointerException during queries. This PR fixes this issue. Specifically: 1. Added null check and initialization for fields in gsonPostProcess 2. Ensured consistent behavior between EditLog replay and Image deserialization 3. Added proper logging for better troubleshooting

…g metadata from image (#47751) cherry-pick (#47603)

Related PR: #47603 Problem Summary: When checking if a table fetched from meta has a remote name, we should use tmpmap

…g metadata from image (apache#47751) cherry-pick (apache#47603)

…rom image (apache#47603) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#41510 Problem Summary: In PR apache#41510, we added several fields to External Catalog. However, we only handled the upgrade scenario for EditLog but not for Image. This causes Catalogs rebuilt from Image to miss these fields, resulting in NullPointerException during queries. This PR fixes this issue. Specifically: 1. Added null check and initialization for fields in gsonPostProcess 2. Ensured consistent behavior between EditLog replay and Image deserialization 3. Added proper logging for better troubleshooting

Related PR: apache#47603 Problem Summary: When checking if a table fetched from meta has a remote name, we should use tmpmap

zy-kkk added dev/2.1.x dev/3.0.x labels Feb 7, 2025

zy-kkk force-pushed the fix_external_npe branch from a683726 to 6311ff2 Compare February 7, 2025 11:48

morningman previously approved these changes Feb 7, 2025

View reviewed changes

morningman added the p0_b label Feb 7, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2025

github-actions bot added the reviewed label Feb 7, 2025

wm1581066 added the usercase Important user case type label label Feb 7, 2025

zy-kkk dismissed morningman’s stale review via 694cb92 February 7, 2025 17:17

zy-kkk force-pushed the fix_external_npe branch from 6311ff2 to 694cb92 Compare February 7, 2025 17:17

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 7, 2025

[fix](external catalog) Fix missing fields when rebuilding metadata f…

76d55b2

…rom image

zy-kkk force-pushed the fix_external_npe branch from 694cb92 to 76d55b2 Compare February 10, 2025 06:34

fix missing set remote db name

232c166

fix

44f9706

fix typo

31db3ec

zy-kkk mentioned this pull request Feb 10, 2025

branch-3.0: [fix](external catalog) Fix missing fields when rebuilding metadata from image #47726

Merged

morningman approved these changes Feb 10, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025

suxiaogang223 approved these changes Feb 10, 2025

View reviewed changes

CalvinKirs approved these changes Feb 10, 2025

View reviewed changes

wuwenchi approved these changes Feb 11, 2025

View reviewed changes

morningman pushed a commit that referenced this pull request Feb 11, 2025

branch-3.0: [fix](external catalog) Fix missing fields when rebuildin…

149c376

…g metadata from image (#47726) cherry-pick #47603

morningman added dev/3.0.4-merged and removed dev/3.0.x labels Feb 11, 2025

morningman merged commit 9f42075 into apache:master Feb 11, 2025
29 of 31 checks passed

github-actions bot added the dev/2.1.x-conflict label Feb 11, 2025

zy-kkk deleted the fix_external_npe branch February 11, 2025 06:50

zy-kkk mentioned this pull request Feb 11, 2025

branch-2.1: [fix](external catalog) Fix missing fields when rebuilding metadata from image #47751

Merged

morningman pushed a commit that referenced this pull request Feb 12, 2025

branch-2.1: [fix](external catalog) Fix missing fields when rebuildin…

da3222e

…g metadata from image (#47751) cherry-pick (#47603)

morningman added dev/2.1.9-merged and removed dev/2.1.x dev/2.1.x-conflict labels Feb 12, 2025

zy-kkk mentioned this pull request Feb 14, 2025

[hotfix](external) Fixed External meta replay check #47928

Merged

16 tasks

zy-kkk added a commit that referenced this pull request Feb 14, 2025

[hotfix](external) Fixed External meta replay check (#47928)

db80419

Related PR: #47603 Problem Summary: When checking if a table fetched from meta has a remote name, we should use tmpmap

morningman pushed a commit to morningman/doris that referenced this pull request Feb 20, 2025

branch-2.1: [fix](external catalog) Fix missing fields when rebuildin…

36cf658

…g metadata from image (apache#47751) cherry-pick (apache#47603)

lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025

[hotfix](external) Fixed External meta replay check (apache#47928)

2e950d6

Related PR: apache#47603 Problem Summary: When checking if a table fetched from meta has a remote name, we should use tmpmap

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](external catalog) Fix missing fields when rebuilding metadata from image #47603

[fix](external catalog) Fix missing fields when rebuilding metadata from image #47603

zy-kkk commented Feb 7, 2025 •

edited

Loading

Thearas commented Feb 7, 2025

zy-kkk commented Feb 7, 2025

morningman left a comment

github-actions bot commented Feb 7, 2025

github-actions bot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

morningman commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

morningman commented Feb 10, 2025

morningman commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

morningman commented Feb 10, 2025

github-actions bot commented Feb 10, 2025

suxiaogang223 left a comment

[fix](external catalog) Fix missing fields when rebuilding metadata from image #47603

[fix](external catalog) Fix missing fields when rebuilding metadata from image #47603

Conversation

zy-kkk commented Feb 7, 2025 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Thearas commented Feb 7, 2025

zy-kkk commented Feb 7, 2025

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 7, 2025

github-actions bot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

morningman commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

morningman commented Feb 10, 2025

morningman commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

doris-robot commented Feb 10, 2025

morningman commented Feb 10, 2025

github-actions bot commented Feb 10, 2025

suxiaogang223 left a comment

Choose a reason for hiding this comment

zy-kkk commented Feb 7, 2025 •

edited

Loading