Skip to content

[feature](agg function) support corr_welford agg function - #49712

Merged
Gabriel39 merged 2 commits into
apache:masterfrom
Mryange:feat-corr-Welford
Apr 21, 2025
Merged

[feature](agg function) support corr_welford agg function#49712
Gabriel39 merged 2 commits into
apache:masterfrom
Mryange:feat-corr-Welford

Conversation

@Mryange

@MryangeMryange commented Mar 31, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

In the past, due to performance considerations, the implementation of corr was relatively simple, avoiding division operations except in the final step.
However, this approach resulted in larger errors, so corr_welford was introduced to improve the accuracy of the calculations.

0.7605339114107809 corr_result
0.7605339114107733 corr_welford_result

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Mar 31, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 92.50% (111/120) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage51.30% (13727/26760)
Line Coverage40.59% (119344/294012)
Region Coverage39.31% (60714/154448)
Branch Coverage34.05% (30426/89344)

@Mryange

Mryange commented Apr 1, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Apr 1, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33912 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92c60770d87fe149f37d6528efbf3b011138d6df, data reload: false
------ Round 1 ----------------------------------
q1	25781	5015	5000	5000
q2	2060	282	162	162
q3	10419	1240	666	666
q4	10223	1007	525	525
q5	7511	2309	2326	2309
q6	185	164	132	132
q7	902	744	599	599
q8	9322	1293	1166	1166
q9	6851	5109	5111	5109
q10	6837	2291	1872	1872
q11	499	278	260	260
q12	348	349	222	222
q13	17778	3612	3056	3056
q14	224	225	208	208
q15	554	488	468	468
q16	637	604	580	580
q17	576	833	344	344
q18	7498	7192	7134	7134
q19	2097	981	560	560
q20	311	328	192	192
q21	3915	2578	2355	2355
q22	1015	1018	993	993
Total cold run time: 115543 ms
Total hot run time: 33912 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5234	5091	5119	5091
q2	228	327	220	220
q3	2148	2645	2215	2215
q4	1411	1810	1354	1354
q5	4423	4375	4440	4375
q6	222	166	129	129
q7	1994	1971	1784	1784
q8	2634	2598	2540	2540
q9	7344	7363	7161	7161
q10	3014	3141	2743	2743
q11	580	531	504	504
q12	667	747	608	608
q13	3500	3872	3295	3295
q14	296	315	306	306
q15	525	483	477	477
q16	650	675	651	651
q17	1160	1568	1346	1346
q18	7805	7556	7404	7404
q19	831	811	843	811
q20	1887	2011	1882	1882
q21	5208	4829	4781	4781
q22	1074	1066	1043	1043
Total cold run time: 52835 ms
Total hot run time: 50720 ms

@doris-robot

doris-robot commented Apr 1, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192217 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92c60770d87fe149f37d6528efbf3b011138d6df, data reload: false
query1	1382	1079	1048	1048
query2	6081	1936	1911	1911
query3	11091	4490	4476	4476
query4	54071	24010	22744	22744
query5	5047	640	468	468
query6	320	193	177	177
query7	4857	511	281	281
query8	308	229	221	221
query9	5478	2574	2579	2574
query10	419	297	259	259
query11	15056	14943	14911	14911
query12	165	113	105	105
query13	1044	514	400	400
query14	10001	6335	6233	6233
query15	203	190	180	180
query16	7047	642	494	494
query17	1106	733	588	588
query18	1566	420	315	315
query19	203	195	182	182
query20	131	131	119	119
query21	212	126	115	115
query22	4418	4520	4324	4324
query23	34064	33146	33296	33146
query24	6640	2467	2427	2427
query25	483	479	410	410
query26	722	272	153	153
query27	2530	519	323	323
query28	3175	2451	2417	2417
query29	591	594	431	431
query30	282	221	186	186
query31	873	900	802	802
query32	75	67	74	67
query33	466	360	324	324
query34	774	855	501	501
query35	813	810	771	771
query36	966	1004	910	910
query37	123	103	81	81
query38	4304	4167	4171	4167
query39	1495	1459	1441	1441
query40	209	115	106	106
query41	54	54	54	54
query42	125	108	108	108
query43	512	520	506	506
query44	1290	801	802	801
query45	182	176	169	169
query46	840	1035	642	642
query47	1886	1927	1782	1782
query48	383	412	314	314
query49	727	532	441	441
query50	707	760	423	423
query51	4227	4309	4273	4273
query52	111	111	97	97
query53	215	260	178	178
query54	482	500	435	435
query55	85	86	84	84
query56	271	270	256	256
query57	1195	1211	1135	1135
query58	263	260	246	246
query59	2724	2862	2748	2748
query60	298	285	271	271
query61	129	130	164	130
query62	738	739	689	689
query63	226	184	182	182
query64	2028	1034	684	684
query65	4443	4318	4328	4318
query66	759	397	290	290
query67	16169	15531	15309	15309
query68	7223	870	511	511
query69	525	289	273	273
query70	1159	1097	1088	1088
query71	490	312	268	268
query72	5920	4815	4889	4815
query73	1504	673	353	353
query74	9240	9034	8976	8976
query75	4040	3233	2736	2736
query76	4271	1181	761	761
query77	725	358	288	288
query78	9959	10136	9275	9275
query79	2269	814	559	559
query80	611	520	443	443
query81	497	255	228	228
query82	467	127	93	93
query83	180	184	159	159
query84	289	96	74	74
query85	878	353	325	325
query86	372	323	283	283
query87	4341	4570	4251	4251
query88	3924	2280	2284	2280
query89	407	313	287	287
query90	1883	209	212	209
query91	157	142	112	112
query92	75	58	61	58
query93	1751	1075	581	581
query94	669	408	312	312
query95	355	279	280	279
query96	491	566	274	274
query97	3192	3232	3129	3129
query98	230	205	202	202
query99	1477	1409	1382	1382
Total cold run time: 299431 ms
Total hot run time: 192217 ms

@doris-robot

doris-robot commented Apr 1, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92c60770d87fe149f37d6528efbf3b011138d6df, data reload: false
query1	0.04	0.04	0.03
query2	0.13	0.10	0.11
query3	0.26	0.19	0.20
query4	1.64	0.19	0.20
query5	0.58	0.57	0.58
query6	1.16	0.71	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.52	0.50
query10	0.58	0.59	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.82	2.70	2.81
query15	0.94	0.84	0.85
query16	0.38	0.38	0.38
query17	1.05	1.01	1.05
query18	0.20	0.19	0.19
query19	1.90	1.94	1.80
query20	0.01	0.01	0.02
query21	15.36	0.92	0.55
query22	0.75	1.04	0.62
query23	15.15	1.40	0.63
query24	6.55	1.97	0.96
query25	0.51	0.18	0.06
query26	0.48	0.17	0.14
query27	0.06	0.05	0.04
query28	9.92	0.81	0.43
query29	12.58	3.97	3.27
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.55	0.47
query33	2.98	3.13	3.12
query34	15.73	5.08	4.49
query35	4.47	4.49	4.51
query36	0.66	0.48	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.22 s
Total hot run time: 31.15 s

@hello-stephen

hello-stephen commented Apr 1, 2025

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 92.50% (111/120) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage51.29% (13726/26760)
Line Coverage40.58% (119304/294012)
Region Coverage39.31% (60707/154448)
Branch Coverage34.04% (30414/89344)

@morningmanmorningman changed the title [feature](agg function) upport corr_welford agg function[feature](agg function) support corr_welford agg functionApr 17, 2025
HappenLee
HappenLee approved these changes Apr 17, 2025
View reviewed changes

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2025
@github-actions

github-actionsBot commented Apr 17, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Apr 17, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

morningman pushed a commit that referenced this pull request Apr 17, 2025
@Mryange@morningman
[feature](agg function) support corr_welford agg function#49712
f60ce5e
@morningmanmorningman added dev/3.0.x usercase Important user case type label and removed dev/3.0.x-experimental labels Apr 21, 2025
@Gabriel39
Gabriel39 merged commit 3aa18fc into apache:masterApr 21, 2025
Mryange added a commit to Mryange/doris that referenced this pull request Apr 22, 2025
@Mryange
[feature](agg function) support corr_welford agg function (apache#49712)
de9f040
In the past, due to performance considerations, the implementation of
corr was relatively simple, avoiding division operations except in the
final step.
However, this approach resulted in larger errors, so corr_welford was
introduced to improve the accuracy of the calculations.
```
0.7605339114107809 corr_result
0.7605339114107733 corr_welford_result
```
dataroaring pushed a commit that referenced this pull request May 14, 2025
@Mryange
[feature](agg function) support corr_welford agg function#49712(#50274
f3a5287
)
In the past, due to performance considerations, the implementation of
corr was relatively simple, avoiding division operations except in the
final step.
However, this approach resulted in larger errors, so corr_welford was
introduced to improve the accuracy of the calculations.
```
0.7605339114107809 corr_result
0.7605339114107733 corr_welford_result
```
@dataroaringdataroaring mentioned this pull request May 14, 2025
Merged
16 tasks
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@Mryangekoarz
[feature](agg function) support corr_welford agg function (apache#49712)
21f0e2a
In the past, due to performance considerations, the implementation of
corr was relatively simple, avoiding division operations except in the
final step.
However, this approach resulted in larger errors, so corr_welford was
introduced to improve the accuracy of the calculations.
```
0.7605339114107809 corr_result
0.7605339114107733 corr_welford_result
```
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HappenLeeHappenLeeHappenLee approved these changes
@Gabriel39Gabriel39Gabriel39 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.6-mergedreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Mryange@hello-stephen@doris-robot@HappenLee@Gabriel39@morningman@dataroaring