[fix](cloud) fix routine load loss data when fe master node restart by sollhui · Pull Request #46149 · apache/doris

sollhui · 2024-12-30T04:06:34Z

What problem does this PR solve?

Problem Summary:

In cloud mode, routine load loss data when fe master node restart.

When updating progress, in order to avoid small values covering large values, we introduced pr #39313, Due to the pr that the routine load replays progress metadata by first obtaining the set default offset and then pulling metadata from meta service to update the local value, if the metadata pulled from meta service is not larger than the set default offset, the correct value cannot be assigned to memory.

To solve this problem, pulling metadata from meta service when restart, determine whether to obtain default offset from Kafka based on the pulled value.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-30T04:06:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2024-12-30T04:06:46Z

run buildall

doris-robot · 2024-12-30T04:45:31Z

TPC-H: Total hot run time: 32467 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b257149b5a6b5739585b905fd98e0601e88e7e2, data reload: false
------ Round 1 ----------------------------------
q1	17572	6124	5995	5995
q2	2040	293	160	160
q3	10456	1213	740	740
q4	10226	872	439	439
q5	7757	2154	1963	1963
q6	200	177	145	145
q7	894	746	596	596
q8	9240	1363	1168	1168
q9	5229	4918	4878	4878
q10	6744	2293	1886	1886
q11	471	282	258	258
q12	335	354	216	216
q13	18003	3587	3018	3018
q14	235	236	223	223
q15	554	510	486	486
q16	636	611	595	595
q17	559	847	321	321
q18	6852	6575	6375	6375
q19	1753	971	534	534
q20	303	315	181	181
q21	2804	2136	1971	1971
q22	359	341	319	319
Total cold run time: 103222 ms
Total hot run time: 32467 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6191	6196	6227	6196
q2	234	322	232	232
q3	2296	2683	2299	2299
q4	1420	1846	1407	1407
q5	4370	4721	4801	4721
q6	185	179	142	142
q7	2041	1946	1774	1774
q8	2590	2829	2666	2666
q9	7376	7232	7358	7232
q10	3060	3352	2872	2872
q11	583	509	500	500
q12	638	745	635	635
q13	3377	3744	3127	3127
q14	287	313	277	277
q15	584	514	500	500
q16	670	702	653	653
q17	1197	1726	1273	1273
q18	7506	7399	7335	7335
q19	811	951	1167	951
q20	1957	2033	1930	1930
q21	5761	5316	4890	4890
q22	619	587	553	553
Total cold run time: 53753 ms
Total hot run time: 52165 ms

doris-robot · 2024-12-30T04:57:54Z

TPC-DS: Total hot run time: 198101 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b257149b5a6b5739585b905fd98e0601e88e7e2, data reload: false
query1	1304	916	913	913
query2	6413	2350	2270	2270
query3	11004	4583	4654	4583
query4	33148	23938	23586	23586
query5	4800	633	464	464
query6	294	210	194	194
query7	3990	500	311	311
query8	302	251	245	245
query9	9295	2743	2751	2743
query10	488	310	276	276
query11	18121	15491	15251	15251
query12	168	105	101	101
query13	1562	542	406	406
query14	10239	7243	8095	7243
query15	228	221	182	182
query16	7828	621	444	444
query17	1500	742	585	585
query18	2085	419	320	320
query19	214	193	186	186
query20	126	123	113	113
query21	200	122	109	109
query22	4608	4739	4575	4575
query23	34401	33647	33756	33647
query24	6475	2363	2321	2321
query25	476	472	419	419
query26	802	271	167	167
query27	2005	474	338	338
query28	5399	2508	2477	2477
query29	590	562	426	426
query30	207	185	153	153
query31	971	934	865	865
query32	73	62	59	59
query33	492	357	335	335
query34	781	860	529	529
query35	833	867	790	790
query36	1023	1065	988	988
query37	119	102	81	81
query38	4229	4141	4266	4141
query39	1728	1570	1475	1475
query40	205	120	103	103
query41	45	44	50	44
query42	126	109	103	103
query43	552	539	515	515
query44	1364	848	851	848
query45	192	177	168	168
query46	904	1067	667	667
query47	1970	1996	1923	1923
query48	390	427	335	335
query49	711	507	398	398
query50	652	652	401	401
query51	7247	7293	7374	7293
query52	111	107	104	104
query53	232	262	188	188
query54	487	503	423	423
query55	80	85	79	79
query56	249	272	266	266
query57	1244	1233	1190	1190
query58	240	227	234	227
query59	3117	3270	3198	3198
query60	289	296	266	266
query61	113	110	114	110
query62	867	806	755	755
query63	229	202	197	197
query64	3109	1055	664	664
query65	3322	3315	3259	3259
query66	756	425	331	331
query67	16491	15867	15582	15582
query68	7935	770	531	531
query69	476	290	250	250
query70	1232	1177	1135	1135
query71	435	295	264	264
query72	6420	4041	3822	3822
query73	657	760	362	362
query74	9903	9275	9284	9275
query75	3953	3182	2672	2672
query76	3601	1183	758	758
query77	769	399	278	278
query78	10114	10072	9475	9475
query79	5334	907	594	594
query80	790	518	543	518
query81	475	269	225	225
query82	656	148	122	122
query83	190	182	149	149
query84	280	97	69	69
query85	805	355	319	319
query86	346	308	298	298
query87	4453	4403	4398	4398
query88	4759	2249	2234	2234
query89	450	335	306	306
query90	1967	193	192	192
query91	135	140	105	105
query92	62	110	54	54
query93	3200	909	543	543
query94	668	384	287	287
query95	332	271	280	271
query96	492	609	290	290
query97	2742	2825	2709	2709
query98	233	202	199	199
query99	1721	1572	1449	1449
Total cold run time: 299400 ms
Total hot run time: 198101 ms

doris-robot · 2024-12-30T05:03:16Z

ClickBench: Total hot run time: 31.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b257149b5a6b5739585b905fd98e0601e88e7e2, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.07
query4	1.61	0.11	0.11
query5	0.42	0.43	0.41
query6	1.15	0.64	0.66
query7	0.02	0.01	0.02
query8	0.03	0.04	0.03
query9	0.59	0.51	0.52
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.60
query14	2.70	2.86	2.73
query15	0.92	0.82	0.84
query16	0.38	0.39	0.39
query17	1.05	1.07	0.99
query18	0.22	0.21	0.21
query19	1.97	1.89	2.05
query20	0.02	0.02	0.01
query21	15.36	0.92	0.59
query22	0.74	0.91	0.64
query23	15.58	1.41	0.53
query24	3.52	1.50	1.37
query25	0.13	0.17	0.13
query26	0.23	0.15	0.14
query27	0.09	0.06	0.05
query28	14.47	1.48	1.04
query29	12.61	3.90	3.20
query30	0.25	0.09	0.06
query31	2.84	0.58	0.38
query32	3.23	0.56	0.46
query33	3.06	3.11	3.08
query34	16.71	5.09	4.52
query35	4.50	4.46	4.45
query36	0.62	0.50	0.48
query37	0.09	0.05	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 107.32 s
Total hot run time: 31.64 s

sollhui · 2024-12-30T13:13:48Z

run buildall

doris-robot · 2024-12-30T13:32:32Z

TPC-H: Total hot run time: 33033 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f44548a0cc5c0f4b93dc6f640b972b9abd6261c3, data reload: false
------ Round 1 ----------------------------------
q1	17600	6189	6111	6111
q2	2052	321	173	173
q3	10391	1277	740	740
q4	10210	865	446	446
q5	7513	2193	2067	2067
q6	213	182	150	150
q7	925	762	622	622
q8	9230	1408	1217	1217
q9	5240	4959	5018	4959
q10	6778	2325	1888	1888
q11	477	278	263	263
q12	359	391	232	232
q13	17755	3553	2958	2958
q14	239	237	222	222
q15	569	509	488	488
q16	632	613	606	606
q17	586	873	324	324
q18	7095	6422	6561	6422
q19	1468	986	588	588
q20	328	328	191	191
q21	2948	2303	2042	2042
q22	369	346	324	324
Total cold run time: 102977 ms
Total hot run time: 33033 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6276	6241	6286	6241
q2	236	328	234	234
q3	2263	2684	2322	2322
q4	1428	1841	1363	1363
q5	4366	4788	4870	4788
q6	201	184	149	149
q7	2138	1979	1799	1799
q8	2594	2831	2668	2668
q9	7335	7178	7184	7178
q10	3119	3338	2869	2869
q11	574	516	493	493
q12	681	752	616	616
q13	3348	3831	3190	3190
q14	272	301	282	282
q15	577	539	487	487
q16	651	679	640	640
q17	1216	1733	1233	1233
q18	7643	7329	7068	7068
q19	817	1199	1068	1068
q20	1931	1997	1919	1919
q21	5518	5223	4919	4919
q22	633	625	567	567
Total cold run time: 53817 ms
Total hot run time: 52093 ms

doris-robot · 2024-12-30T13:44:41Z

TPC-DS: Total hot run time: 191287 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f44548a0cc5c0f4b93dc6f640b972b9abd6261c3, data reload: false
query1	978	397	387	387
query2	6516	2407	2344	2344
query3	6701	215	214	214
query4	33969	23455	23368	23368
query5	4371	686	462	462
query6	321	199	200	199
query7	4628	533	305	305
query8	314	259	258	258
query9	9587	2757	2750	2750
query10	482	321	248	248
query11	18378	15462	15175	15175
query12	161	109	106	106
query13	1677	572	424	424
query14	9835	7552	7407	7407
query15	261	196	186	186
query16	8126	654	422	422
query17	1576	771	590	590
query18	2119	457	295	295
query19	225	184	151	151
query20	120	109	113	109
query21	211	127	107	107
query22	4472	4389	4382	4382
query23	34659	33471	33549	33471
query24	6620	2299	2254	2254
query25	478	455	396	396
query26	1206	293	153	153
query27	2118	472	326	326
query28	5352	2481	2456	2456
query29	731	562	428	428
query30	230	190	148	148
query31	996	920	822	822
query32	82	63	60	60
query33	517	362	290	290
query34	787	881	512	512
query35	823	819	737	737
query36	1024	1099	956	956
query37	130	105	82	82
query38	4263	4129	4104	4104
query39	1517	1469	1437	1437
query40	208	117	101	101
query41	48	43	47	43
query42	126	106	104	104
query43	529	521	492	492
query44	1343	804	798	798
query45	180	179	165	165
query46	931	1122	648	648
query47	1896	1936	1850	1850
query48	380	405	321	321
query49	791	501	385	385
query50	649	707	383	383
query51	7174	7185	7146	7146
query52	104	105	93	93
query53	237	262	194	194
query54	488	483	416	416
query55	79	81	78	78
query56	254	260	243	243
query57	1195	1184	1144	1144
query58	237	221	242	221
query59	3142	3301	2753	2753
query60	290	265	257	257
query61	115	114	117	114
query62	876	813	729	729
query63	234	191	195	191
query64	4430	1010	648	648
query65	3322	3180	3224	3180
query66	1051	441	311	311
query67	15841	15868	15506	15506
query68	8933	779	528	528
query69	470	302	254	254
query70	1207	1148	1120	1120
query71	428	294	275	275
query72	5809	3982	3970	3970
query73	660	813	366	366
query74	9848	9033	8924	8924
query75	4497	3209	2668	2668
query76	5337	1331	874	874
query77	992	374	298	298
query78	10120	10271	9655	9655
query79	4059	937	581	581
query80	720	551	440	440
query81	472	278	228	228
query82	636	155	127	127
query83	199	170	158	158
query84	287	88	73	73
query85	837	358	310	310
query86	351	329	312	312
query87	4542	4615	4463	4463
query88	3377	2258	2217	2217
query89	458	338	309	309
query90	1999	192	190	190
query91	138	144	104	104
query92	64	59	56	56
query93	2150	926	541	541
query94	669	432	288	288
query95	344	271	253	253
query96	487	665	286	286
query97	2782	2845	2747	2747
query98	218	219	202	202
query99	1705	1582	1435	1435
Total cold run time: 297506 ms
Total hot run time: 191287 ms

doris-robot · 2024-12-30T13:50:00Z

ClickBench: Total hot run time: 31.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f44548a0cc5c0f4b93dc6f640b972b9abd6261c3, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.44	0.39	0.42
query6	1.17	0.65	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.52	0.50
query10	0.55	0.58	0.57
query11	0.15	0.10	0.10
query12	0.14	0.12	0.12
query13	0.61	0.61	0.60
query14	2.77	2.88	2.77
query15	0.90	0.83	0.82
query16	0.37	0.38	0.38
query17	0.96	0.99	1.01
query18	0.23	0.21	0.20
query19	1.95	1.81	2.01
query20	0.01	0.01	0.01
query21	15.36	0.96	0.58
query22	0.76	0.91	0.60
query23	15.28	1.41	0.63
query24	3.02	0.90	1.17
query25	0.19	0.16	0.10
query26	0.22	0.14	0.13
query27	0.08	0.06	0.04
query28	14.08	1.54	1.04
query29	12.61	3.93	3.26
query30	0.25	0.09	0.06
query31	2.82	0.60	0.38
query32	3.22	0.54	0.46
query33	3.07	3.09	3.07
query34	16.90	5.10	4.53
query35	4.56	4.53	4.51
query36	0.65	0.52	0.48
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.4 s
Total hot run time: 31.23 s

liaoxin01

LGTM

github-actions · 2025-01-02T09:58:06Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-02T09:58:09Z

PR approved by anyone and no changes requested.

MoanasDaddyXu

LGTM

…46149) In cloud mode, routine load loss data when fe master node restart. When updating progress, in order to avoid small values covering large values, we introduced pr #39313, Due to the pr that the routine load replays progress metadata by first obtaining the set default offset and then pulling metadata from meta service to update the local value, if the metadata pulled from meta service is not larger than the set default offset, the correct value cannot be assigned to memory. To solve this problem, pulling metadata from meta service when restart, determine whether to obtain default offset from Kafka based on the pulled value.

…de restart #46149 (#46566) Cherry-picked from #46149 Co-authored-by: hui lai <laihui@selectdb.com>

sollhui force-pushed the fix_rl_loss_data branch from dfb56df to 548d59bCompare December 30, 2024 12:20

fix routine load loss data when fe master node restart
f44548a

sollhui force-pushed the fix_rl_loss_data branch from 548d59b to f44548aCompare December 30, 2024 13:13

dataroaring added dev/3.0.x usercase Important user case type label labels Dec 31, 2024

liaoxin01 approved these changes Jan 2, 2025
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 2, 2025

MoanasDaddyXu approved these changes Jan 6, 2025
View reviewed changes

dataroaring merged commit ed07dc8 into apache:masterJan 7, 2025

github-actionsBot mentioned this pull request Jan 7, 2025
branch-3.0: [fix](cloud) fix routine load loss data when fe master node restart #46149 #46566
Merged

liaoxin01 pushed a commit that referenced this pull request Jan 8, 2025
branch-3.0: [fix](cloud) fix routine load loss data when fe master no…
197f860
…de restart #46149 (#46566) Cherry-picked from #46149 Co-authored-by: hui lai <laihui@selectdb.com>

liaoxin01 added dev/3.0.4-merged and removed dev/3.0.x labels Jan 8, 2025

Navigation Menu

Uh oh!

[fix](cloud) fix routine load loss data when fe master node restart - #46149

[fix](cloud) fix routine load loss data when fe master node restart#46149
dataroaring merged 1 commit into
apache:masterfrom
sollhui:fix_rl_loss_data

sollhui commented Dec 30, 2024•
edited by liaoxin01
Loading

Uh oh!

Thearas commented Dec 30, 2024

Uh oh!

sollhui commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

sollhui commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

liaoxin01 left a comment

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

MoanasDaddyXu left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

sollhui commented Dec 30, 2024• edited by liaoxin01 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 30, 2024

Uh oh!

sollhui commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

sollhui commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

MoanasDaddyXu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sollhui commented Dec 30, 2024•
edited by liaoxin01
Loading