[fix](gson) Missing the serialization of the partition's storage policy by wyxxxcat · Pull Request #49721 · apache/doris

wyxxxcat · 2025-04-01T01:56:55Z

What problem does this PR solve?

partition's storage_policy are lost when the fe master sync with the follower or when the fe recovers

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-01T01:57:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wyxxxcat · 2025-04-01T09:11:50Z

run buildall

doris-robot · 2025-04-01T09:56:03Z

TPC-H: Total hot run time: 34312 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 090d0bf458bb228aa931b96b75c2119785aad5bf, data reload: false
------ Round 1 ----------------------------------
q1	25716	5438	5088	5088
q2	2070	282	191	191
q3	10450	1226	691	691
q4	10562	1026	555	555
q5	7553	2301	2353	2301
q6	184	163	134	134
q7	899	746	612	612
q8	9321	1254	1122	1122
q9	6846	5165	5134	5134
q10	6845	2274	1924	1924
q11	496	276	278	276
q12	346	356	221	221
q13	17788	3707	3129	3129
q14	228	222	209	209
q15	528	482	500	482
q16	624	612	576	576
q17	600	862	376	376
q18	7548	7107	7112	7107
q19	1744	963	554	554
q20	326	337	237	237
q21	4090	3350	2425	2425
q22	1043	1026	968	968
Total cold run time: 115807 ms
Total hot run time: 34312 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5227	5086	5121	5086
q2	235	329	243	243
q3	2121	2615	2272	2272
q4	1422	1828	1430	1430
q5	4462	4377	4346	4346
q6	215	170	133	133
q7	1990	1874	1748	1748
q8	2608	2532	2501	2501
q9	7203	7209	7121	7121
q10	2978	3151	2743	2743
q11	580	512	489	489
q12	683	755	591	591
q13	3571	3886	3336	3336
q14	281	307	260	260
q15	525	483	480	480
q16	637	672	644	644
q17	1132	1565	1365	1365
q18	7611	7406	7548	7406
q19	803	823	838	823
q20	1924	1963	1810	1810
q21	5333	4854	4784	4784
q22	1122	1034	1059	1034
Total cold run time: 52663 ms
Total hot run time: 50645 ms

doris-robot · 2025-04-01T10:07:53Z

TPC-DS: Total hot run time: 192611 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 090d0bf458bb228aa931b96b75c2119785aad5bf, data reload: false
query1	1401	1059	1033	1033
query2	6406	1908	1868	1868
query3	11156	4628	4572	4572
query4	25452	23630	23135	23135
query5	4155	621	463	463
query6	298	206	192	192
query7	3993	492	281	281
query8	289	236	237	236
query9	8517	2555	2583	2555
query10	487	313	261	261
query11	15556	14994	14847	14847
query12	163	109	110	109
query13	1556	524	399	399
query14	8710	6083	6293	6083
query15	199	188	172	172
query16	7167	646	471	471
query17	1189	753	559	559
query18	1972	424	313	313
query19	196	195	156	156
query20	141	122	116	116
query21	203	127	113	113
query22	4557	4454	4444	4444
query23	34292	33383	33275	33275
query24	8520	2490	2464	2464
query25	511	469	407	407
query26	1160	271	155	155
query27	2815	496	344	344
query28	4922	2441	2403	2403
query29	725	584	452	452
query30	288	224	204	204
query31	934	897	803	803
query32	74	71	63	63
query33	572	379	330	330
query34	805	872	532	532
query35	824	836	752	752
query36	992	1032	963	963
query37	126	110	81	81
query38	4195	4279	4111	4111
query39	1467	1449	1432	1432
query40	213	127	108	108
query41	54	52	82	52
query42	120	108	109	108
query43	500	523	479	479
query44	1304	816	824	816
query45	183	173	176	173
query46	872	1044	650	650
query47	1825	1884	1817	1817
query48	381	415	314	314
query49	728	510	426	426
query50	663	699	417	417
query51	4244	4495	4338	4338
query52	108	107	102	102
query53	230	255	190	190
query54	581	579	519	519
query55	83	94	88	88
query56	324	331	294	294
query57	1161	1193	1112	1112
query58	270	260	278	260
query59	2725	2857	2658	2658
query60	328	332	325	325
query61	140	133	134	133
query62	780	738	694	694
query63	241	191	193	191
query64	4209	1053	706	706
query65	4428	4456	4387	4387
query66	1120	405	313	313
query67	16617	15569	15193	15193
query68	8272	887	518	518
query69	481	296	267	267
query70	1193	1081	1111	1081
query71	454	306	311	306
query72	5114	4782	4711	4711
query73	668	588	352	352
query74	8918	9140	8758	8758
query75	3933	3228	2731	2731
query76	3763	1185	770	770
query77	787	366	298	298
query78	9843	9887	9300	9300
query79	1864	830	562	562
query80	649	525	442	442
query81	460	255	218	218
query82	242	131	98	98
query83	265	251	230	230
query84	286	107	85	85
query85	823	354	314	314
query86	339	309	289	289
query87	4401	4385	4283	4283
query88	2808	2251	2245	2245
query89	383	313	285	285
query90	2027	217	221	217
query91	154	141	115	115
query92	75	64	58	58
query93	1121	935	596	596
query94	652	417	292	292
query95	380	310	347	310
query96	476	558	278	278
query97	3093	3239	3115	3115
query98	222	202	203	202
query99	1382	1415	1273	1273
Total cold run time: 276945 ms
Total hot run time: 192611 ms

doris-robot · 2025-04-01T10:13:10Z

ClickBench: Total hot run time: 31.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 090d0bf458bb228aa931b96b75c2119785aad5bf, data reload: false
query1	0.04	0.04	0.02
query2	0.12	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.20
query5	0.59	0.58	0.57
query6	1.20	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.51
query10	0.57	0.58	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.70	2.69	2.71
query15	0.92	0.86	0.84
query16	0.38	0.38	0.38
query17	1.00	0.98	1.02
query18	0.22	0.19	0.20
query19	1.91	1.83	1.89
query20	0.01	0.01	0.02
query21	15.34	0.89	0.53
query22	0.76	1.09	0.65
query23	15.07	1.37	0.63
query24	6.55	1.90	0.98
query25	0.53	0.23	0.07
query26	0.58	0.17	0.14
query27	0.05	0.05	0.04
query28	9.45	0.87	0.42
query29	12.51	3.92	3.26
query30	0.25	0.09	0.07
query31	2.86	0.59	0.38
query32	3.22	0.54	0.47
query33	3.05	3.11	3.09
query34	15.86	5.11	4.52
query35	4.51	4.56	4.52
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.02
Total cold run time: 104.79 s
Total hot run time: 31.23 s

dataroaring

LGTM

github-actions · 2025-04-02T03:07:00Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-02T03:07:02Z

PR approved by anyone and no changes requested.

wyxxxcat · 2025-04-02T08:07:34Z

run buildall

doris-robot · 2025-04-02T08:47:27Z

TPC-H: Total hot run time: 33843 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c54a3f1d4ea877672ef1eae8ae7ddb96f29aad4b, data reload: false
------ Round 1 ----------------------------------
q1	25802	5128	4948	4948
q2	2075	268	182	182
q3	10398	1247	695	695
q4	10238	1001	525	525
q5	7546	2285	2325	2285
q6	179	161	133	133
q7	911	739	608	608
q8	9329	1297	1066	1066
q9	6934	5098	5074	5074
q10	6792	2292	1888	1888
q11	486	284	277	277
q12	351	345	226	226
q13	17765	3655	3069	3069
q14	234	224	204	204
q15	538	505	478	478
q16	619	618	569	569
q17	618	850	378	378
q18	7520	7031	7070	7031
q19	1226	939	553	553
q20	334	344	236	236
q21	4061	3490	2454	2454
q22	1069	1024	964	964
Total cold run time: 115025 ms
Total hot run time: 33843 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5102	5125	5070	5070
q2	240	327	228	228
q3	2142	2613	2277	2277
q4	1385	1795	1353	1353
q5	4442	4531	4379	4379
q6	214	167	132	132
q7	2005	1901	1788	1788
q8	2582	2575	2485	2485
q9	7262	7298	7033	7033
q10	2988	3212	2746	2746
q11	577	506	492	492
q12	671	756	617	617
q13	3522	3913	3281	3281
q14	309	305	276	276
q15	524	485	476	476
q16	645	683	655	655
q17	1142	1558	1385	1385
q18	7823	7494	7412	7412
q19	827	820	909	820
q20	1897	1954	1843	1843
q21	5256	4921	4822	4822
q22	1102	1055	1041	1041
Total cold run time: 52657 ms
Total hot run time: 50611 ms

doris-robot · 2025-04-02T08:59:17Z

TPC-DS: Total hot run time: 192152 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c54a3f1d4ea877672ef1eae8ae7ddb96f29aad4b, data reload: false
query1	1404	1064	1026	1026
query2	6323	1983	1918	1918
query3	11121	4515	4617	4515
query4	25535	23749	23127	23127
query5	4292	601	444	444
query6	309	215	190	190
query7	3986	540	274	274
query8	292	240	231	231
query9	8498	2570	2542	2542
query10	482	313	264	264
query11	15258	15147	14719	14719
query12	161	111	107	107
query13	1565	527	402	402
query14	8690	6068	6136	6068
query15	200	186	164	164
query16	7190	638	474	474
query17	1194	707	548	548
query18	1970	407	312	312
query19	198	202	162	162
query20	131	120	115	115
query21	203	123	107	107
query22	4419	4531	4396	4396
query23	33888	33259	33213	33213
query24	8724	2461	2401	2401
query25	557	465	402	402
query26	1272	274	152	152
query27	2850	512	339	339
query28	4750	2427	2437	2427
query29	760	596	477	477
query30	277	230	203	203
query31	914	883	785	785
query32	79	69	66	66
query33	573	389	322	322
query34	818	907	529	529
query35	788	844	768	768
query36	948	1002	910	910
query37	126	106	85	85
query38	4185	4375	4056	4056
query39	1506	1420	1510	1420
query40	219	121	111	111
query41	54	55	52	52
query42	120	106	110	106
query43	512	538	482	482
query44	1311	811	807	807
query45	179	176	167	167
query46	846	1028	649	649
query47	1812	1870	1785	1785
query48	381	415	312	312
query49	790	528	435	435
query50	651	704	408	408
query51	4259	4253	4271	4253
query52	113	110	103	103
query53	239	267	184	184
query54	596	588	522	522
query55	85	77	80	77
query56	325	297	280	280
query57	1181	1185	1142	1142
query58	265	260	263	260
query59	2761	2961	2712	2712
query60	362	343	349	343
query61	147	129	129	129
query62	788	753	695	695
query63	229	188	189	188
query64	4123	1055	701	701
query65	4404	4345	4328	4328
query66	1110	410	314	314
query67	16396	15503	15678	15503
query68	8436	875	515	515
query69	480	313	256	256
query70	1120	1104	1082	1082
query71	478	321	287	287
query72	5278	4707	4708	4707
query73	675	586	356	356
query74	8817	8959	8767	8767
query75	3843	3211	2737	2737
query76	3671	1188	751	751
query77	877	367	290	290
query78	9765	10062	9218	9218
query79	1954	810	552	552
query80	629	520	453	453
query81	460	255	220	220
query82	245	131	99	99
query83	266	254	227	227
query84	287	109	90	90
query85	750	348	314	314
query86	327	286	302	286
query87	4464	4580	4341	4341
query88	2819	2190	2190	2190
query89	405	311	281	281
query90	2000	208	208	208
query91	135	144	113	113
query92	73	57	54	54
query93	1849	937	574	574
query94	657	409	310	310
query95	374	294	285	285
query96	495	555	278	278
query97	3078	3223	3113	3113
query98	225	201	202	201
query99	1364	1432	1289	1289
Total cold run time: 276990 ms
Total hot run time: 192152 ms

doris-robot · 2025-04-02T09:04:35Z

ClickBench: Total hot run time: 31.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c54a3f1d4ea877672ef1eae8ae7ddb96f29aad4b, data reload: false
query1	0.03	0.04	0.03
query2	0.12	0.10	0.10
query3	0.24	0.20	0.20
query4	1.60	0.20	0.19
query5	0.59	0.58	0.59
query6	1.18	0.72	0.70
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.51
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.61	0.59	0.60
query14	2.75	2.70	2.79
query15	0.93	0.86	0.86
query16	0.40	0.39	0.38
query17	1.09	1.04	1.02
query18	0.21	0.19	0.20
query19	1.87	1.96	1.84
query20	0.02	0.02	0.01
query21	15.38	0.91	0.55
query22	0.75	1.07	0.93
query23	14.78	1.42	0.61
query24	7.18	0.83	1.01
query25	0.48	0.14	0.09
query26	0.60	0.16	0.13
query27	0.05	0.06	0.06
query28	9.91	0.90	0.44
query29	12.56	3.96	3.24
query30	0.26	0.10	0.08
query31	2.80	0.59	0.38
query32	3.23	0.54	0.47
query33	2.98	3.05	3.14
query34	15.60	5.15	4.50
query35	4.52	4.58	4.51
query36	0.69	0.48	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.44 s
Total hot run time: 31.41 s

dataroaring

LGTM

github-actions · 2025-04-07T01:36:47Z

PR approved by at least one committer and no changes requested.

dataroaring

LGTM

…cy (#49721) partition's `storage_policy` are lost when the fe master sync with the follower or when the fe recovers

…storage policy #49721 (#49840) Cherry-picked from #49721 Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

…storage policy #49721 (#49841) Cherry-picked from #49721 Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

…cy (apache#49721) partition's `storage_policy` are lost when the fe master sync with the follower or when the fe recovers

wyxxxcat force-pushed the sp_miss_Serialized branch from 834070c to 090d0bfCompare April 1, 2025 09:11

dataroaring previously approved these changes Apr 2, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2025

github-actionsBot added the reviewed label Apr 2, 2025

dataroaring added dev/2.1.x dev/3.0.x labels Apr 2, 2025

yagagagaga approved these changes Apr 2, 2025
View reviewed changes

wyxxxcat marked this pull request as draft April 2, 2025 07:54

1
c54a3f1

wyxxxcat dismissed dataroaring’s stale review via c54a3f1April 2, 2025 08:07

wyxxxcat force-pushed the sp_miss_Serialized branch from 090d0bf to c54a3f1Compare April 2, 2025 08:07

wyxxxcat marked this pull request as ready for review April 2, 2025 08:07

wyxxxcat requested review from dataroaring and yagagagaga April 2, 2025 08:07

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Apr 2, 2025

yagagagaga approved these changes Apr 3, 2025
View reviewed changes

dataroaring approved these changes Apr 7, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 7, 2025

dataroaring approved these changes Apr 7, 2025
View reviewed changes

dataroaring merged commit 15934a7 into apache:masterApr 7, 2025

github-actionsBot pushed a commit that referenced this pull request Apr 7, 2025
[fix](gson) Missing the serialization of the partition's storage poli…
2ac2f46
…cy (#49721) partition's `storage_policy` are lost when the fe master sync with the follower or when the fe recovers

github-actionsBot mentioned this pull request Apr 7, 2025
branch-3.0: [fix](gson) Missing the serialization of the partition's storage policy #49721 #49840
Merged

github-actionsBot pushed a commit that referenced this pull request Apr 7, 2025
[fix](gson) Missing the serialization of the partition's storage poli…
2b02948
…cy (#49721) partition's `storage_policy` are lost when the fe master sync with the follower or when the fe recovers

github-actionsBot mentioned this pull request Apr 7, 2025
branch-2.1: [fix](gson) Missing the serialization of the partition's storage policy #49721 #49841
Merged

dataroaring pushed a commit that referenced this pull request Apr 7, 2025
branch-3.0: [fix](gson) Missing the serialization of the partition's …
64cfdf7
…storage policy #49721 (#49840) Cherry-picked from #49721 Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Apr 7, 2025

yiguolei pushed a commit that referenced this pull request Apr 8, 2025
branch-2.1: [fix](gson) Missing the serialization of the partition's …
11fecd1
…storage policy #49721 (#49841) Cherry-picked from #49721 Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

yiguolei added dev/2.1.10-merged and removed dev/2.1.x labels Apr 8, 2025

yiguolei mentioned this pull request May 13, 2025
2.1.10 Release Note #50853
Closed

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[fix](gson) Missing the serialization of the partition's storage poli…
33930f6
…cy (apache#49721) partition's `storage_policy` are lost when the fe master sync with the follower or when the fe recovers

Uh oh!

Conversation

wyxxxcat commented Apr 1, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 1, 2025

Uh oh!

wyxxxcat commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

wyxxxcat commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

doris-robot commented Apr 2, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 7, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants