Skip to content

[Opt](partial update) Add some cases for partial update - #47900

Merged
dataroaring merged 7 commits into
apache:masterfrom
bobhan1:add-case-for-pu
Feb 20, 2025
Merged

[Opt](partial update) Add some cases for partial update#47900
dataroaring merged 7 commits into
apache:masterfrom
bobhan1:add-case-for-pu

Conversation

@bobhan1

@bobhan1bobhan1 commented Feb 14, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

add some cases for partial update

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1

bobhan1 commented Feb 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31433 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c57b2e77681ea3b7df4c50b403cf24cf8ebb6d2b, data reload: false
------ Round 1 ----------------------------------
q1	17594	5211	5065	5065
q2	2051	299	168	168
q3	10405	1341	706	706
q4	10219	1141	535	535
q5	7528	2342	2387	2342
q6	191	176	140	140
q7	891	741	606	606
q8	9287	1293	1107	1107
q9	4857	4581	4755	4581
q10	6814	2296	1879	1879
q11	470	280	263	263
q12	349	358	229	229
q13	17767	3718	3044	3044
q14	223	222	206	206
q15	501	454	464	454
q16	624	610	575	575
q17	561	865	353	353
q18	6707	6158	6169	6158
q19	1082	949	546	546
q20	300	329	190	190
q21	2674	2195	1975	1975
q22	371	328	311	311
Total cold run time: 101466 ms
Total hot run time: 31433 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5090	5089	5090	5089
q2	240	326	230	230
q3	2176	2662	2303	2303
q4	1465	1833	1376	1376
q5	4201	4157	4147	4147
q6	205	166	127	127
q7	1882	1831	1667	1667
q8	2594	2563	2564	2563
q9	7326	7136	7060	7060
q10	3058	3211	2796	2796
q11	569	513	482	482
q12	698	801	648	648
q13	3475	3888	3262	3262
q14	285	302	281	281
q15	507	471	463	463
q16	641	665	645	645
q17	1135	1615	1309	1309
q18	7683	7349	7235	7235
q19	774	817	998	817
q20	1926	2018	1862	1862
q21	5543	5038	4864	4864
q22	628	594	539	539
Total cold run time: 52101 ms
Total hot run time: 49765 ms

@doris-robot

doris-robot commented Feb 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 183047 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c57b2e77681ea3b7df4c50b403cf24cf8ebb6d2b, data reload: false
query1	964	378	358	358
query2	6514	1924	1912	1912
query3	6788	219	212	212
query4	26451	23485	23402	23402
query5	4349	662	496	496
query6	293	206	178	178
query7	4612	491	299	299
query8	293	227	221	221
query9	8608	2482	2499	2482
query10	451	323	256	256
query11	15710	14968	14833	14833
query12	156	116	107	107
query13	1659	540	400	400
query14	9622	6030	6106	6030
query15	202	180	168	168
query16	7317	632	473	473
query17	1165	679	531	531
query18	1950	387	296	296
query19	197	183	152	152
query20	119	118	115	115
query21	240	121	102	102
query22	4405	4515	4238	4238
query23	33870	33072	32844	32844
query24	8187	2395	2387	2387
query25	506	447	379	379
query26	1238	265	153	153
query27	2722	467	336	336
query28	4350	2395	2384	2384
query29	745	530	423	423
query30	233	181	162	162
query31	942	866	756	756
query32	71	63	64	63
query33	571	360	308	308
query34	782	850	523	523
query35	826	812	734	734
query36	945	948	895	895
query37	119	92	79	79
query38	4178	4211	4116	4116
query39	1489	1397	1382	1382
query40	205	111	103	103
query41	53	50	52	50
query42	123	107	105	105
query43	503	501	479	479
query44	1270	791	796	791
query45	181	167	164	164
query46	856	1051	654	654
query47	1802	1822	1743	1743
query48	374	403	304	304
query49	776	515	421	421
query50	670	741	430	430
query51	4139	4213	4102	4102
query52	112	117	93	93
query53	219	253	185	185
query54	469	484	409	409
query55	80	85	81	81
query56	273	256	254	254
query57	1120	1151	1087	1087
query58	236	237	271	237
query59	2667	2785	2672	2672
query60	311	285	281	281
query61	145	138	136	136
query62	800	724	651	651
query63	229	200	198	198
query64	4370	1094	736	736
query65	3177	3177	3119	3119
query66	1108	424	303	303
query67	16053	15464	15227	15227
query68	7397	761	511	511
query69	464	300	263	263
query70	1152	1104	1092	1092
query71	396	308	261	261
query72	5719	3575	3673	3575
query73	743	730	359	359
query74	8882	8988	8681	8681
query75	3165	3163	2699	2699
query76	3216	1175	732	732
query77	481	357	274	274
query78	9893	10064	9278	9278
query79	2593	795	606	606
query80	646	531	492	492
query81	528	269	237	237
query82	195	125	100	100
query83	190	170	157	157
query84	236	90	78	78
query85	734	335	301	301
query86	365	307	276	276
query87	4534	4412	4386	4386
query88	3911	2208	2196	2196
query89	397	316	294	294
query90	1978	198	198	198
query91	137	139	110	110
query92	72	61	55	55
query93	2193	1033	586	586
query94	682	400	298	298
query95	366	278	260	260
query96	475	561	274	274
query97	2758	2866	2735	2735
query98	234	206	195	195
query99	1282	1414	1288	1288
Total cold run time: 272174 ms
Total hot run time: 183047 ms

@doris-robot

doris-robot commented Feb 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.19 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c57b2e77681ea3b7df4c50b403cf24cf8ebb6d2b, data reload: false
query1	0.04	0.03	0.05
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.42	0.43	0.39
query6	1.15	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.58	0.51	0.55
query10	0.58	0.57	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.61
query14	2.69	2.71	2.69
query15	0.93	0.85	0.87
query16	0.38	0.37	0.36
query17	1.02	1.01	1.02
query18	0.22	0.20	0.20
query19	1.90	1.85	1.98
query20	0.01	0.01	0.02
query21	15.35	0.90	0.54
query22	0.76	1.20	0.65
query23	14.94	1.40	0.65
query24	11.82	1.18	0.41
query25	0.32	0.25	0.13
query26	0.92	0.18	0.14
query27	0.04	0.04	0.04
query28	5.96	0.76	0.44
query29	12.57	3.89	3.24
query30	0.26	0.08	0.07
query31	2.83	0.59	0.40
query32	3.21	0.55	0.48
query33	2.99	3.05	2.99
query34	15.87	5.19	4.54
query35	4.54	4.52	4.55
query36	0.65	0.48	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.32 s
Total hot run time: 30.19 s

@bobhan1

bobhan1 commented Feb 17, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 17, 2025

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 43.80% (11442/26121)
Line Coverage: 33.80% (96354/285050)
Region Coverage: 32.52% (49295/151597)
Branch Coverage: 28.18% (24768/87880)
Coverage Report: http://coverage.selectdb-in.cc/coverage/154acf064023d0a2588894bdc873d56e9cb204f0_154acf064023d0a2588894bdc873d56e9cb204f0/report/index.html

@bobhan1

bobhan1 commented Feb 18, 2025

Copy link
Copy Markdown
ContributorAuthor

run p0

@bobhan1

bobhan1 commented Feb 18, 2025

Copy link
Copy Markdown
ContributorAuthor

run cloud_p0

zhannngchen
zhannngchen approved these changes Feb 19, 2025
View reviewed changes

@zhannngchenzhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Feb 19, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 19, 2025
@github-actions

github-actionsBot commented Feb 19, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@bobhan1

bobhan1 commented Feb 19, 2025

Copy link
Copy Markdown
ContributorAuthor

run performance

@doris-robot

doris-robot commented Feb 19, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31410 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 154acf064023d0a2588894bdc873d56e9cb204f0, data reload: false
------ Round 1 ----------------------------------
q1	17648	5321	5041	5041
q2	2051	300	188	188
q3	10390	1292	734	734
q4	10218	1019	560	560
q5	7491	2434	2307	2307
q6	188	167	133	133
q7	917	753	595	595
q8	9317	1383	1186	1186
q9	5032	4749	4667	4667
q10	6851	2328	1870	1870
q11	485	272	255	255
q12	349	359	215	215
q13	17774	3667	3110	3110
q14	235	221	212	212
q15	499	461	463	461
q16	621	601	572	572
q17	573	853	336	336
q18	7050	6318	6071	6071
q19	1209	949	510	510
q20	305	326	193	193
q21	2798	2171	1889	1889
q22	364	323	305	305
Total cold run time: 102365 ms
Total hot run time: 31410 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5081	5092	5082	5082
q2	233	321	225	225
q3	2177	2690	2326	2326
q4	1473	1838	1375	1375
q5	4192	4115	4102	4102
q6	201	166	122	122
q7	1866	1824	1628	1628
q8	2603	2541	2526	2526
q9	7338	7058	7177	7058
q10	3040	3155	2715	2715
q11	576	500	493	493
q12	697	778	629	629
q13	3537	3877	3174	3174
q14	279	296	268	268
q15	507	468	447	447
q16	653	671	631	631
q17	1106	1616	1327	1327
q18	7409	7436	7400	7400
q19	755	787	930	787
q20	2010	2047	1849	1849
q21	5436	4980	4819	4819
q22	612	562	557	557
Total cold run time: 51781 ms
Total hot run time: 49540 ms

@doris-robot

doris-robot commented Feb 19, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 183762 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 154acf064023d0a2588894bdc873d56e9cb204f0, data reload: false
query1	972	388	380	380
query2	6533	1887	1850	1850
query3	6809	214	211	211
query4	26604	23706	23432	23432
query5	4353	660	495	495
query6	306	204	206	204
query7	4621	505	293	293
query8	290	241	235	235
query9	8647	2557	2550	2550
query10	487	329	262	262
query11	15510	15019	14921	14921
query12	161	106	106	106
query13	1660	540	393	393
query14	9972	6230	6334	6230
query15	212	191	177	177
query16	7651	608	469	469
query17	1191	715	561	561
query18	1985	401	300	300
query19	216	202	158	158
query20	121	123	117	117
query21	227	121	104	104
query22	4254	4232	4520	4232
query23	34556	33498	32957	32957
query24	8117	2355	2382	2355
query25	514	458	391	391
query26	1222	262	149	149
query27	2657	498	329	329
query28	4236	2412	2403	2403
query29	722	537	417	417
query30	230	184	153	153
query31	953	853	796	796
query32	69	69	60	60
query33	553	351	305	305
query34	760	842	535	535
query35	787	799	735	735
query36	948	975	908	908
query37	120	97	76	76
query38	4215	4153	4104	4104
query39	1449	1403	1388	1388
query40	206	110	100	100
query41	53	54	52	52
query42	124	103	107	103
query43	529	515	477	477
query44	1297	819	817	817
query45	173	173	160	160
query46	858	1034	645	645
query47	1751	1811	1744	1744
query48	383	403	311	311
query49	781	487	415	415
query50	681	730	426	426
query51	4176	4235	4079	4079
query52	104	104	91	91
query53	229	251	192	192
query54	477	489	447	447
query55	84	81	77	77
query56	263	268	244	244
query57	1142	1149	1054	1054
query58	252	225	239	225
query59	2550	2613	2501	2501
query60	283	277	259	259
query61	119	114	118	114
query62	768	758	656	656
query63	231	192	189	189
query64	4392	970	656	656
query65	3244	3153	3131	3131
query66	1049	402	297	297
query67	15889	15767	15383	15383
query68	5987	770	520	520
query69	461	300	268	268
query70	1193	1147	1112	1112
query71	406	289	277	277
query72	5719	3592	3660	3592
query73	739	757	357	357
query74	9285	9175	8738	8738
query75	3211	3135	2721	2721
query76	3247	1171	731	731
query77	475	362	282	282
query78	10051	10023	9409	9409
query79	2739	845	620	620
query80	641	530	447	447
query81	519	282	259	259
query82	513	126	99	99
query83	170	178	153	153
query84	239	95	73	73
query85	819	351	304	304
query86	376	312	298	298
query87	4446	4462	4386	4386
query88	4308	2235	2229	2229
query89	400	312	285	285
query90	1926	195	197	195
query91	140	141	108	108
query92	75	63	56	56
query93	2317	1024	590	590
query94	703	401	305	305
query95	353	281	270	270
query96	486	559	295	295
query97	2807	2856	2735	2735
query98	241	209	202	202
query99	1328	1437	1260	1260
Total cold run time: 273298 ms
Total hot run time: 183762 ms

@doris-robot

doris-robot commented Feb 19, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 154acf064023d0a2588894bdc873d56e9cb204f0, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.40	0.41	0.39
query6	1.17	0.67	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.55
query10	0.57	0.57	0.57
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	2.70	2.70	2.71
query15	0.92	0.86	0.84
query16	0.40	0.37	0.36
query17	1.00	1.01	1.01
query18	0.21	0.19	0.20
query19	1.87	1.76	1.92
query20	0.02	0.01	0.02
query21	15.35	0.89	0.54
query22	0.78	1.07	0.64
query23	15.10	1.40	0.64
query24	11.48	1.27	0.37
query25	0.39	0.23	0.08
query26	0.87	0.18	0.14
query27	0.05	0.05	0.05
query28	6.13	0.78	0.43
query29	12.57	3.88	3.25
query30	0.25	0.09	0.07
query31	2.81	0.58	0.38
query32	3.24	0.55	0.47
query33	3.08	3.06	3.03
query34	15.89	5.14	4.51
query35	4.59	4.60	4.57
query36	0.65	0.50	0.48
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.51 s
Total hot run time: 30.02 s

dataroaring
dataroaring approved these changes Feb 20, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit c1f9a1b into apache:masterFeb 20, 2025
github-actionsBot pushed a commit that referenced this pull request Feb 20, 2025
@bobhan1Your Name
[Opt](partial update) Add some cases for partial update (#47900)
49968e5
add some cases for partial update
@github-actionsgithub-actionsBot mentioned this pull request Feb 20, 2025
Merged
dataroaring pushed a commit that referenced this pull request Feb 20, 2025
@github-actions@bobhan1
branch-3.0: [Opt](partial update) Add some cases for partial update#…
2180229
…47900 (#48129)
Cherry-picked from #47900
Co-authored-by: bobhan1 <baohan@selectdb.com>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@bobhan1@lzyy2024
[Opt](partial update) Add some cases for partial update (apache#47900)
53a98e0
add some cases for partial update
bobhan1 added a commit to bobhan1/doris that referenced this pull request Feb 25, 2025
@bobhan1
[Opt](partial update) Add some cases for partial update (apache#47900)
946a6e1
add some cases for partial update
@bobhan1bobhan1 mentioned this pull request Feb 25, 2025
Merged
bobhan1 added a commit to bobhan1/doris that referenced this pull request Feb 25, 2025
@bobhan1
[Opt](partial update) Add some cases for partial update (apache#47900)
8858303
add some cases for partial update
yiguolei pushed a commit that referenced this pull request Feb 26, 2025
@bobhan1
branch-2.1-pick: [Opt](partial update) Add some cases for partial upd…
45ebb21
…ate (#47900) (#48272)
pick #47900
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@bobhan1koarz
[Opt](partial update) Add some cases for partial update (apache#47900)
b351291
add some cases for partial update
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchenzhannngchenzhannngchen approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.9-mergeddev/3.0.5-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@bobhan1@Thearas@doris-robot@zhannngchen@dataroaring@yiguolei