Skip to content

[fix](nereids) fix bucket shuffle join to right compute wrong output property - #47888

Merged
924060929 merged 6 commits into
apache:masterfrom
924060929:fix_shuffle_to_right
Feb 17, 2025
Merged

[fix](nereids) fix bucket shuffle join to right compute wrong output property#47888
924060929 merged 6 commits into
apache:masterfrom
924060929:fix_shuffle_to_right

Conversation

@924060929

@924060929924060929 commented Feb 13, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

fix Illegal bucket shuffle join or colocate join in fragment because compute wrong join output property, introduced by #41730

the exception:

errCode = 2, detailMessage = Illegal bucket shuffle join or colocate join in fragment

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Feb 13, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929

924060929 commented Feb 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@924060929

924060929 commented Feb 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@924060929

924060929 commented Feb 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 13, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31747 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f682e6368aab052211d1b03455586e871f6d046b, data reload: false
------ Round 1 ----------------------------------
q1	17808	5203	5223	5203
q2	2060	292	165	165
q3	10431	1220	740	740
q4	10212	988	536	536
q5	7543	2343	2335	2335
q6	189	169	135	135
q7	902	758	597	597
q8	9309	1237	1041	1041
q9	4892	4776	4688	4688
q10	6849	2329	1933	1933
q11	454	274	252	252
q12	360	365	234	234
q13	17810	3726	3081	3081
q14	224	219	211	211
q15	503	476	457	457
q16	619	612	588	588
q17	556	865	336	336
q18	6486	6251	6222	6222
q19	1495	926	539	539
q20	308	328	184	184
q21	2667	2216	1968	1968
q22	363	329	302	302
Total cold run time: 102040 ms
Total hot run time: 31747 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5091	5028	5084	5028
q2	232	321	231	231
q3	2163	2654	2267	2267
q4	1382	1803	1358	1358
q5	4257	4139	4193	4139
q6	208	161	124	124
q7	1847	1814	1627	1627
q8	2557	2602	2516	2516
q9	7146	7173	7098	7098
q10	2982	3191	2752	2752
q11	560	525	498	498
q12	683	800	638	638
q13	3561	3849	3324	3324
q14	283	295	274	274
q15	511	464	451	451
q16	655	680	640	640
q17	1127	1603	1324	1324
q18	7589	7481	7420	7420
q19	805	823	849	823
q20	1951	2006	1886	1886
q21	5399	4901	4792	4792
q22	621	588	532	532
Total cold run time: 51610 ms
Total hot run time: 49742 ms

@doris-robot

doris-robot commented Feb 13, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 189419 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f682e6368aab052211d1b03455586e871f6d046b, data reload: false
query1	1303	939	943	939
query2	6242	1808	1843	1808
query3	10990	4364	4260	4260
query4	54517	25375	22977	22977
query5	5205	605	473	473
query6	342	197	211	197
query7	4916	499	289	289
query8	329	238	230	230
query9	5485	2557	2572	2557
query10	437	309	251	251
query11	15186	14967	14818	14818
query12	161	110	110	110
query13	1056	504	381	381
query14	10349	6899	6305	6305
query15	197	201	178	178
query16	7081	664	481	481
query17	1103	746	593	593
query18	1562	416	325	325
query19	212	192	167	167
query20	137	134	130	130
query21	212	125	110	110
query22	4508	4636	4437	4437
query23	33915	33132	33178	33132
query24	5846	2443	2403	2403
query25	460	477	403	403
query26	733	275	151	151
query27	1900	479	353	353
query28	2836	2486	2432	2432
query29	574	564	426	426
query30	220	190	155	155
query31	912	858	834	834
query32	79	65	62	62
query33	443	359	308	308
query34	783	851	525	525
query35	809	852	795	795
query36	962	999	915	915
query37	125	98	83	83
query38	4161	4289	4278	4278
query39	1570	1463	1434	1434
query40	199	116	105	105
query41	51	49	77	49
query42	131	103	99	99
query43	508	520	474	474
query44	1320	822	807	807
query45	188	174	173	173
query46	872	1055	652	652
query47	1839	1858	1796	1796
query48	389	428	310	310
query49	758	509	415	415
query50	716	751	415	415
query51	4263	4280	4267	4267
query52	111	102	101	101
query53	241	257	184	184
query54	484	485	415	415
query55	85	81	94	81
query56	287	281	281	281
query57	1142	1165	1136	1136
query58	253	245	236	236
query59	2880	2906	2819	2819
query60	310	270	274	270
query61	126	118	160	118
query62	747	740	669	669
query63	228	186	187	186
query64	1830	1004	666	666
query65	3216	3143	3156	3143
query66	772	392	295	295
query67	16060	15554	15524	15524
query68	4763	768	505	505
query69	502	304	264	264
query70	1210	1100	1147	1100
query71	428	293	272	272
query72	5988	3780	3718	3718
query73	816	719	346	346
query74	9051	8960	8721	8721
query75	3179	3123	2693	2693
query76	3745	1169	748	748
query77	530	363	363	363
query78	10047	10110	9337	9337
query79	2045	807	592	592
query80	716	535	457	457
query81	507	272	244	244
query82	410	126	95	95
query83	163	167	154	154
query84	287	89	68	68
query85	751	359	298	298
query86	348	301	266	266
query87	4395	4465	4324	4324
query88	2784	2228	2181	2181
query89	394	316	286	286
query90	1593	189	188	188
query91	136	139	113	113
query92	60	56	58	56
query93	1775	993	576	576
query94	663	378	297	297
query95	343	260	250	250
query96	477	549	270	270
query97	2785	2839	2772	2772
query98	239	207	198	198
query99	1284	1409	1276	1276
Total cold run time: 290278 ms
Total hot run time: 189419 ms

@doris-robot

doris-robot commented Feb 13, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f682e6368aab052211d1b03455586e871f6d046b, data reload: false
query1	0.03	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.41	0.41	0.40
query6	1.14	0.65	0.67
query7	0.02	0.01	0.02
query8	0.03	0.04	0.03
query9	0.60	0.52	0.52
query10	0.56	0.59	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.70	2.69	2.70
query15	0.91	0.86	0.86
query16	0.37	0.38	0.39
query17	1.03	1.06	1.02
query18	0.22	0.19	0.20
query19	1.93	1.84	2.02
query20	0.01	0.01	0.01
query21	15.35	0.91	0.54
query22	0.76	1.22	0.81
query23	14.71	1.40	0.62
query24	8.16	4.14	0.59
query25	0.33	0.10	0.07
query26	0.91	0.19	0.13
query27	0.06	0.05	0.05
query28	6.00	0.77	0.43
query29	12.56	3.85	3.24
query30	0.25	0.08	0.06
query31	2.83	0.57	0.38
query32	3.22	0.54	0.47
query33	3.04	3.02	3.05
query34	15.78	5.15	4.49
query35	4.54	4.50	4.52
query36	0.68	0.50	0.48
query37	0.09	0.06	0.07
query38	0.06	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 102.5 s
Total hot run time: 30.41 s

@924060929

924060929 commented Feb 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@924060929

924060929 commented Feb 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31704 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 969e8188974aa4094d687a074ba5d24b9d76ab31, data reload: false
------ Round 1 ----------------------------------
q1	17585	5199	5093	5093
q2	2047	311	168	168
q3	10401	1256	742	742
q4	10254	1024	543	543
q5	7900	2413	2371	2371
q6	185	166	133	133
q7	901	751	602	602
q8	9299	1308	1106	1106
q9	4956	4830	4659	4659
q10	6823	2285	1893	1893
q11	474	274	252	252
q12	340	366	219	219
q13	17779	3678	3093	3093
q14	231	239	209	209
q15	512	480	467	467
q16	609	624	573	573
q17	582	882	338	338
q18	6506	6246	6328	6246
q19	1990	1101	560	560
q20	302	317	183	183
q21	2798	2121	1948	1948
q22	373	333	306	306
Total cold run time: 102847 ms
Total hot run time: 31704 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5281	5141	5146	5141
q2	240	335	228	228
q3	2137	2690	2337	2337
q4	1417	1828	1379	1379
q5	4257	4150	4178	4150
q6	205	162	124	124
q7	1868	1823	1751	1751
q8	2628	2607	2618	2607
q9	7232	7170	7069	7069
q10	3016	3223	2807	2807
q11	577	517	504	504
q12	697	752	655	655
q13	3499	3878	3275	3275
q14	277	312	257	257
q15	516	481	479	479
q16	643	691	630	630
q17	1136	1625	1314	1314
q18	7667	7495	7463	7463
q19	824	817	837	817
q20	2016	2053	1901	1901
q21	5428	4989	4760	4760
q22	638	567	555	555
Total cold run time: 52199 ms
Total hot run time: 50203 ms

@doris-robot

doris-robot commented Feb 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 189288 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 969e8188974aa4094d687a074ba5d24b9d76ab31, data reload: false
query1	1329	958	924	924
query2	6320	1840	1831	1831
query3	10967	4286	4380	4286
query4	54023	24355	22991	22991
query5	5292	578	470	470
query6	362	179	183	179
query7	5107	529	295	295
query8	307	263	221	221
query9	6691	2494	2489	2489
query10	426	301	265	265
query11	15297	15101	14776	14776
query12	164	106	106	106
query13	1159	527	385	385
query14	10774	6790	6205	6205
query15	214	197	180	180
query16	7088	694	510	510
query17	1057	714	546	546
query18	1526	413	305	305
query19	203	194	159	159
query20	128	119	124	119
query21	205	119	103	103
query22	4389	4477	4291	4291
query23	34103	33322	33299	33299
query24	5626	2423	2405	2405
query25	457	478	398	398
query26	692	295	155	155
query27	1782	544	341	341
query28	2763	2445	2414	2414
query29	580	559	451	451
query30	223	193	159	159
query31	894	883	823	823
query32	82	65	65	65
query33	460	361	313	313
query34	785	905	506	506
query35	810	843	750	750
query36	967	980	926	926
query37	128	103	74	74
query38	4256	4304	4331	4304
query39	1485	1463	1420	1420
query40	208	117	107	107
query41	51	50	53	50
query42	135	102	102	102
query43	503	519	484	484
query44	1379	832	826	826
query45	180	172	164	164
query46	887	1085	661	661
query47	1862	1864	1792	1792
query48	395	421	308	308
query49	710	529	424	424
query50	760	743	410	410
query51	4260	4349	4214	4214
query52	110	105	100	100
query53	232	259	191	191
query54	495	489	418	418
query55	85	78	81	78
query56	263	264	278	264
query57	1164	1220	1136	1136
query58	247	236	242	236
query59	2752	2726	2552	2552
query60	290	275	283	275
query61	119	113	118	113
query62	742	744	690	690
query63	238	200	190	190
query64	1720	1085	748	748
query65	3202	3198	3125	3125
query66	731	413	304	304
query67	15806	15511	15393	15393
query68	4816	797	575	575
query69	499	296	271	271
query70	1194	1030	1069	1030
query71	437	298	256	256
query72	6315	3756	3947	3756
query73	853	810	354	354
query74	8868	9029	9065	9029
query75	3236	3134	2690	2690
query76	3582	1237	794	794
query77	534	381	280	280
query78	9991	10066	9320	9320
query79	2556	829	606	606
query80	673	566	449	449
query81	509	282	231	231
query82	294	131	93	93
query83	180	177	148	148
query84	288	91	75	75
query85	757	348	301	301
query86	346	312	305	305
query87	4474	4547	4420	4420
query88	3641	2226	2200	2200
query89	405	321	298	298
query90	1808	194	199	194
query91	134	145	110	110
query92	70	61	56	56
query93	2541	1024	582	582
query94	686	418	295	295
query95	348	272	261	261
query96	501	604	275	275
query97	2768	2858	2783	2783
query98	229	208	207	207
query99	1314	1430	1250	1250
Total cold run time: 293494 ms
Total hot run time: 189288 ms

@doris-robot

doris-robot commented Feb 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 969e8188974aa4094d687a074ba5d24b9d76ab31, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.42	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.51	0.52
query10	0.56	0.58	0.58
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.60
query14	2.83	2.75	2.69
query15	0.93	0.86	0.86
query16	0.38	0.39	0.36
query17	1.05	1.02	1.02
query18	0.21	0.19	0.20
query19	1.91	1.83	1.97
query20	0.01	0.00	0.02
query21	15.36	0.88	0.56
query22	0.76	1.14	0.96
query23	14.73	1.39	0.61
query24	12.09	1.02	0.34
query25	0.34	0.12	0.08
query26	0.55	0.18	0.14
query27	0.04	0.05	0.06
query28	6.15	0.77	0.44
query29	12.56	3.99	3.33
query30	0.25	0.08	0.06
query31	2.84	0.59	0.38
query32	3.24	0.56	0.48
query33	3.05	2.96	3.04
query34	15.92	5.12	4.53
query35	4.54	4.53	4.51
query36	0.70	0.50	0.50
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.02	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.58 s
Total hot run time: 30.41 s

@github-actions

github-actionsBot commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2025
@github-actions

github-actionsBot commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

xzj7019
xzj7019 reviewed Feb 14, 2025
View reviewed changes
Comment threadregression-test/suites/nereids_syntax_p0/distribute/bucket_shuffle_to_right.groovy

suite("bucket_shuffle_to_right") {
multi_sql """
set enable_nereids_distribute_planner=true;

@xzj7019xzj7019Feb 14, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this case have problem if new coordinator is off?

@924060929924060929Feb 14, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this case have problem if new coordinator is off?

no, only new coordinator wrong

@924060929
924060929 merged commit 2a2786f into apache:masterFeb 17, 2025
@924060929
924060929 deleted the fix_shuffle_to_right branch February 19, 2025 09:13
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@924060929@lzyy2024
[fix](nereids) fix bucket shuffle join to right compute wrong output …
a739c6b
…property (apache#47888)
fix Illegal bucket shuffle join or colocate join in fragment because
compute wrong join output property, introduced by apache#41730
the exception:
```
errCode = 2, detailMessage = Illegal bucket shuffle join or colocate join in fragment
```
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@924060929koarz
[fix](nereids) fix bucket shuffle join to right compute wrong output …
5bfa920
…property (apache#47888)
fix Illegal bucket shuffle join or colocate join in fragment because
compute wrong join output property, introduced by apache#41730
the exception:
```
errCode = 2, detailMessage = Illegal bucket shuffle join or colocate join in fragment
```
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowmorrySnow approved these changes
@xzj7019xzj7019xzj7019 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@924060929@Thearas@doris-robot@morrySnow@xzj7019