[chore](binlog) Record rollup index info for alterJob binlog by wyxxxcat · Pull Request #50337 · apache/doris

wyxxxcat · 2025-04-23T09:33:02Z

What problem does this PR solve?

Issue Number: close #xxx

ccr: selectdb/ccr-syncer#601

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-04-23T09:33:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wyxxxcat · 2025-04-23T09:34:42Z

run buildall

doris-robot · 2025-04-23T09:56:13Z

TeamCity cloud ut coverage result:
Function Coverage: 83.06% (1098/1322)
Line Coverage: 65.83% (18365/27896)
Region Coverage: 65.24% (9096/13942)
Branch Coverage: 55.20% (4892/8862)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3acbdcfd0bfea9e7ddc905d94a759704af53b2cd_3acbdcfd0bfea9e7ddc905d94a759704af53b2cd_cloud/report/index.html

doris-robot · 2025-04-23T11:18:37Z

TPC-H: Total hot run time: 33786 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3acbdcfd0bfea9e7ddc905d94a759704af53b2cd, data reload: false
------ Round 1 ----------------------------------
q1	26138	5154	5017	5017
q2	2086	272	194	194
q3	10400	1259	699	699
q4	10251	1028	525	525
q5	8034	2414	2361	2361
q6	181	161	131	131
q7	922	733	608	608
q8	9317	1261	1070	1070
q9	6743	5041	5077	5041
q10	6851	2275	1896	1896
q11	471	274	264	264
q12	348	358	220	220
q13	17795	3682	3099	3099
q14	241	227	208	208
q15	533	495	498	495
q16	454	460	401	401
q17	581	851	349	349
q18	7741	7205	7026	7026
q19	2829	960	556	556
q20	315	328	212	212
q21	3780	3378	2424	2424
q22	1007	1032	990	990
Total cold run time: 117018 ms
Total hot run time: 33786 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5239	5078	5048	5048
q2	236	323	234	234
q3	2159	2660	2285	2285
q4	1436	1845	1519	1519
q5	4507	4509	4460	4460
q6	224	169	123	123
q7	2010	1954	1768	1768
q8	2605	2539	2617	2539
q9	7266	7101	7221	7101
q10	2988	3186	2756	2756
q11	576	518	491	491
q12	709	767	626	626
q13	3499	3865	3341	3341
q14	272	294	261	261
q15	529	481	465	465
q16	466	509	469	469
q17	1179	1554	1434	1434
q18	7677	7584	7407	7407
q19	792	806	891	806
q20	1979	1978	1864	1864
q21	5559	4927	4937	4927
q22	1063	1040	1034	1034
Total cold run time: 52970 ms
Total hot run time: 50958 ms

doris-robot · 2025-04-23T11:30:51Z

TPC-DS: Total hot run time: 192882 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3acbdcfd0bfea9e7ddc905d94a759704af53b2cd, data reload: false
query1	1431	1097	1079	1079
query2	6176	1847	1844	1844
query3	10950	4542	4466	4466
query4	54924	25706	23244	23244
query5	5123	584	447	447
query6	345	200	191	191
query7	4957	503	280	280
query8	311	237	232	232
query9	5844	2557	2609	2557
query10	440	317	269	269
query11	15068	14984	14892	14892
query12	161	106	105	105
query13	1076	504	389	389
query14	10963	6385	6386	6385
query15	210	194	183	183
query16	7055	680	502	502
query17	1099	758	596	596
query18	1588	428	320	320
query19	198	202	182	182
query20	129	127	123	123
query21	214	165	108	108
query22	4245	4494	4328	4328
query23	34189	33391	33562	33391
query24	5669	2426	2438	2426
query25	475	473	391	391
query26	669	278	155	155
query27	1828	477	335	335
query28	3082	2166	2147	2147
query29	565	568	441	441
query30	271	224	193	193
query31	859	851	808	808
query32	75	65	67	65
query33	475	382	314	314
query34	876	848	537	537
query35	799	833	775	775
query36	945	990	908	908
query37	120	102	80	80
query38	4137	4304	4338	4304
query39	1492	1442	1458	1442
query40	222	124	109	109
query41	55	57	83	57
query42	120	115	114	114
query43	518	527	484	484
query44	1314	833	842	833
query45	190	177	172	172
query46	855	1045	644	644
query47	1831	1844	1788	1788
query48	380	418	334	334
query49	723	546	429	429
query50	641	683	402	402
query51	4263	4281	4216	4216
query52	113	107	104	104
query53	234	254	178	178
query54	576	575	499	499
query55	84	82	82	82
query56	309	283	277	277
query57	1137	1206	1137	1137
query58	263	254	252	252
query59	2746	2761	2696	2696
query60	330	323	308	308
query61	133	129	132	129
query62	742	751	657	657
query63	221	194	182	182
query64	1554	1195	802	802
query65	4344	4211	4227	4211
query66	750	407	324	324
query67	15798	15605	15366	15366
query68	6812	889	515	515
query69	529	310	255	255
query70	1142	1149	1086	1086
query71	515	315	285	285
query72	5687	4728	4974	4728
query73	1285	661	352	352
query74	9065	8901	8921	8901
query75	4086	3187	2701	2701
query76	4266	1194	741	741
query77	760	364	278	278
query78	10185	10117	9309	9309
query79	2955	805	571	571
query80	672	513	434	434
query81	474	262	223	223
query82	620	126	100	100
query83	247	248	229	229
query84	294	111	81	81
query85	817	360	324	324
query86	370	304	275	275
query87	4349	4412	4387	4387
query88	3571	2220	2163	2163
query89	404	315	280	280
query90	1805	216	215	215
query91	153	143	113	113
query92	67	66	58	58
query93	2436	955	578	578
query94	660	420	309	309
query95	374	306	287	287
query96	473	576	274	274
query97	3176	3185	3140	3140
query98	237	214	202	202
query99	1433	1419	1268	1268
Total cold run time: 299398 ms
Total hot run time: 192882 ms

doris-robot · 2025-04-23T11:36:08Z

ClickBench: Total hot run time: 30.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3acbdcfd0bfea9e7ddc905d94a759704af53b2cd, data reload: false
query1	0.04	0.04	0.02
query2	0.15	0.11	0.10
query3	0.35	0.20	0.20
query4	1.59	0.20	0.20
query5	0.61	0.61	0.61
query6	1.18	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.61	0.51	0.51
query10	0.58	0.59	0.57
query11	0.26	0.13	0.13
query12	0.25	0.13	0.13
query13	0.65	0.62	0.62
query14	1.18	1.20	1.20
query15	0.95	0.88	0.86
query16	0.38	0.38	0.37
query17	1.08	1.02	1.02
query18	0.19	0.18	0.18
query19	1.96	1.85	1.80
query20	0.01	0.01	0.01
query21	15.39	0.98	0.68
query22	0.91	0.97	0.84
query23	14.71	1.52	0.75
query24	5.43	0.56	0.28
query25	0.18	0.09	0.09
query26	0.54	0.21	0.18
query27	0.09	0.08	0.09
query28	11.05	1.14	0.56
query29	12.53	4.22	3.47
query30	0.28	0.09	0.06
query31	2.82	0.63	0.43
query32	3.24	0.59	0.50
query33	3.00	3.13	3.11
query34	16.50	5.08	4.37
query35	4.41	4.45	4.48
query36	0.64	0.50	0.49
query37	0.21	0.16	0.17
query38	0.17	0.16	0.16
query39	0.04	0.04	0.04
query40	0.20	0.15	0.17
query41	0.11	0.05	0.05
query42	0.06	0.05	0.06
query43	0.06	0.05	0.05
Total cold run time: 104.66 s
Total hot run time: 30.31 s

doris-robot · 2025-04-23T12:58:11Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.53% (14538/27157)
Line Coverage	42.35% (126018/297576)
Region Coverage	41.17% (64408/156446)
Branch Coverage	35.75% (32377/90562)

wyxxxcat · 2025-04-23T14:44:23Z

run buildall

doris-robot · 2025-04-23T15:06:47Z

TPC-H: Total hot run time: 33976 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db47e44c75437b5431bf47bd846c94c61755595e, data reload: false
------ Round 1 ----------------------------------
q1	26259	5059	5052	5052
q2	2096	275	181	181
q3	10410	1219	695	695
q4	10235	1017	537	537
q5	7530	2352	2346	2346
q6	176	163	131	131
q7	914	750	614	614
q8	9333	1328	1065	1065
q9	6868	5043	5056	5043
q10	6817	2319	1901	1901
q11	479	290	270	270
q12	361	353	226	226
q13	17784	3734	3119	3119
q14	229	231	217	217
q15	538	501	498	498
q16	452	449	395	395
q17	575	859	358	358
q18	7514	7114	7133	7114
q19	1234	934	568	568
q20	340	339	243	243
q21	4163	3377	2442	2442
q22	1034	1006	961	961
Total cold run time: 115341 ms
Total hot run time: 33976 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5140	5119	5163	5119
q2	238	327	228	228
q3	2135	2675	2271	2271
q4	1415	1816	1421	1421
q5	4577	4414	4358	4358
q6	210	167	125	125
q7	1948	1863	1722	1722
q8	2605	2501	2476	2476
q9	7162	7034	7159	7034
q10	2969	3171	2705	2705
q11	560	500	487	487
q12	693	770	643	643
q13	3532	3922	3304	3304
q14	285	289	283	283
q15	560	487	495	487
q16	465	502	473	473
q17	1176	1542	1392	1392
q18	7698	7414	7383	7383
q19	807	800	949	800
q20	1972	2051	1862	1862
q21	5245	4978	4583	4583
q22	1053	1021	982	982
Total cold run time: 52445 ms
Total hot run time: 50138 ms

doris-robot · 2025-04-23T15:18:14Z

TPC-DS: Total hot run time: 185034 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db47e44c75437b5431bf47bd846c94c61755595e, data reload: false
query1	1008	493	501	493
query2	6557	1783	1787	1783
query3	6741	226	214	214
query4	26969	23738	22932	22932
query5	4365	621	455	455
query6	320	214	202	202
query7	4635	481	286	286
query8	328	253	241	241
query9	8629	2587	2594	2587
query10	473	343	272	272
query11	15341	14996	14761	14761
query12	148	107	101	101
query13	1652	515	411	411
query14	8789	6041	6023	6023
query15	196	182	165	165
query16	7117	643	477	477
query17	937	705	553	553
query18	1990	394	292	292
query19	193	187	159	159
query20	124	141	118	118
query21	209	120	104	104
query22	4136	4113	4022	4022
query23	33775	32973	33178	32973
query24	8417	2367	2396	2367
query25	534	453	382	382
query26	1231	264	166	166
query27	2743	493	335	335
query28	4347	2128	2092	2092
query29	754	549	426	426
query30	283	214	183	183
query31	915	832	765	765
query32	75	66	61	61
query33	562	356	321	321
query34	795	902	511	511
query35	766	804	711	711
query36	971	1020	898	898
query37	120	107	75	75
query38	4062	4144	4206	4144
query39	1490	1409	1585	1409
query40	210	119	105	105
query41	57	57	52	52
query42	122	107	114	107
query43	499	493	461	461
query44	1296	796	800	796
query45	178	169	177	169
query46	830	999	621	621
query47	1744	1784	1726	1726
query48	374	398	325	325
query49	773	504	429	429
query50	622	671	407	407
query51	4046	4079	4074	4074
query52	109	103	101	101
query53	219	249	193	193
query54	581	572	498	498
query55	80	77	77	77
query56	317	292	276	276
query57	1123	1157	1088	1088
query58	259	248	246	246
query59	2509	2573	2492	2492
query60	320	314	312	312
query61	128	124	156	124
query62	777	749	653	653
query63	225	189	189	189
query64	4392	1114	773	773
query65	4341	4260	4259	4259
query66	1174	414	341	341
query67	15643	15590	15309	15309
query68	7776	867	516	516
query69	469	297	271	271
query70	1218	1112	1089	1089
query71	428	316	287	287
query72	5632	4707	4710	4707
query73	632	695	343	343
query74	8847	9088	8644	8644
query75	3427	3193	2718	2718
query76	3239	1180	749	749
query77	595	355	291	291
query78	9895	10084	9336	9336
query79	2023	847	552	552
query80	618	504	442	442
query81	475	256	230	230
query82	475	127	92	92
query83	242	251	230	230
query84	291	103	89	89
query85	874	358	312	312
query86	333	318	284	284
query87	4430	4382	4314	4314
query88	3520	2210	2213	2210
query89	376	309	288	288
query90	1872	214	214	214
query91	142	135	113	113
query92	73	60	57	57
query93	1394	940	577	577
query94	726	407	288	288
query95	367	286	282	282
query96	473	549	276	276
query97	3100	3205	3094	3094
query98	233	213	196	196
query99	1338	1397	1270	1270
Total cold run time: 271178 ms
Total hot run time: 185034 ms

doris-robot · 2025-04-23T15:23:27Z

ClickBench: Total hot run time: 29.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db47e44c75437b5431bf47bd846c94c61755595e, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.12
query3	0.25	0.20	0.19
query4	1.59	0.20	0.19
query5	0.58	0.58	0.60
query6	1.18	0.72	0.72
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.59	0.54	0.52
query10	0.57	0.56	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	1.18	1.21	1.17
query15	0.87	0.85	0.84
query16	0.39	0.39	0.37
query17	1.04	1.02	1.01
query18	0.22	0.20	0.20
query19	1.87	1.83	1.83
query20	0.01	0.01	0.01
query21	15.44	0.89	0.54
query22	0.78	1.05	0.69
query23	14.97	1.39	0.63
query24	6.79	1.31	1.10
query25	0.50	0.20	0.07
query26	0.62	0.17	0.13
query27	0.05	0.05	0.05
query28	10.29	0.90	0.44
query29	12.59	4.00	3.34
query30	0.25	0.08	0.06
query31	2.82	0.58	0.39
query32	3.22	0.56	0.46
query33	3.03	3.09	3.05
query34	15.77	5.07	4.47
query35	4.52	4.50	4.46
query36	0.68	0.50	0.48
query37	0.09	0.07	0.07
query38	0.06	0.04	0.03
query39	0.03	0.03	0.02
query40	0.18	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 104.31 s
Total hot run time: 29.88 s

wyxxxcat · 2025-04-23T15:40:14Z

run feut

dataroaring

LGTM

github-actions · 2025-04-24T07:05:53Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-24T07:05:55Z

PR approved by anyone and no changes requested.

wyxxxcat · 2025-04-25T02:06:58Z

run feut

koarz

LGTM

Hastyshell

LGTM

…50337) ccr: selectdb/ccr-syncer#601

…#50850, #50337 (#50873) cherry pick from #50850, #50337 --------- Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

…#50850, #50337 (#50874) cherry pick from #50850, #50337 --------- Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

…50337) ccr: selectdb/ccr-syncer#601

wyxxxcat requested review from CalvinKirs, dataroaring and morningman as code owners April 23, 2025 09:33

wyxxxcat mentioned this pull request Apr 23, 2025
test: apply add rollup before upsert to avoid full sync selectdb/ccr-syncer#601
Merged

1
db47e44

wyxxxcat force-pushed the record_add_index branch from 3acbdcf to db47e44Compare April 23, 2025 14:43

wyxxxcat changed the title ~~[chore](binlog) GetMeta returns added indexes and commit seq~~[chore](binlog) Record rollup index info for alterJob binlogApr 23, 2025

dataroaring approved these changes Apr 24, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 24, 2025

github-actionsBot added the reviewed label Apr 24, 2025

koarz approved these changes Apr 25, 2025
View reviewed changes

Hastyshell approved these changes Apr 25, 2025
View reviewed changes

dataroaring merged commit 535070f into apache:masterApr 25, 2025

w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request May 14, 2025
[chore](binlog) Record rollup index info for alterJob binlog (apache#…
e5aedeb
…50337) ccr: selectdb/ccr-syncer#601

w41ter mentioned this pull request May 14, 2025
branch-2.1: [fix](binlog) Record rollup index info for alterJob binlog #50850, #50337 #50874
Merged

w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request May 14, 2025
[chore](binlog) Record rollup index info for alterJob binlog (apache#…
cf338fc
…50337) ccr: selectdb/ccr-syncer#601

w41ter mentioned this pull request May 14, 2025
branch-3.0: [fix](binlog) Record rollup index info for alterJob binlog #50850, #50337 #50873
Merged

dataroaring pushed a commit that referenced this pull request May 14, 2025
branch-3.0: [fix](binlog) Record rollup index info for alterJob binlog…
4d60b97
…#50850, #50337 (#50873) cherry pick from #50850, #50337 --------- Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

dataroaring added the dev/3.0.6-merged label May 14, 2025

yiguolei pushed a commit that referenced this pull request May 17, 2025
branch-2.1: [fix](binlog) Record rollup index info for alterJob binlog…
35af34f
…#50850, #50337 (#50874) cherry pick from #50850, #50337 --------- Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>

yiguolei added the dev/2.1.11-merged label May 17, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[chore](binlog) Record rollup index info for alterJob binlog (apache#…
2ba4c65
…50337) ccr: selectdb/ccr-syncer#601

gavinchou mentioned this pull request Jun 11, 2025
Release Note 3.0.6 #51585
Closed

Uh oh!

Conversation

wyxxxcat commented Apr 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Apr 23, 2025

Uh oh!

wyxxxcat commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

BE UT Coverage Report

Uh oh!

wyxxxcat commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

wyxxxcat commented Apr 23, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 24, 2025

Uh oh!

github-actionsBot commented Apr 24, 2025

Uh oh!

wyxxxcat commented Apr 25, 2025

Uh oh!

koarz left a comment

Choose a reason for hiding this comment

Uh oh!

Hastyshell left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

wyxxxcat commented Apr 23, 2025•
edited
Loading