[feat](nereids)using salt-join automatically when encountering skew join by englefly · Pull Request #54207 · apache/doris

englefly · 2025-08-01T08:41:41Z

What problem does this PR solve?

When encountering a data-skewed join, there are currently two optimization methods: using salt-join or using broadcast join.
If we detect data skew during the RBO phase and the right table is relatively large, we will automatically add salt.
If no salt is automatically added during the RBO phase, we will try to choose broadcast join based on cost during the CBO phase.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-08-01T08:41:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2025-08-05T02:08:51Z

run buildall

doris-robot · 2025-08-05T02:40:46Z

TPC-H: Total hot run time: 33512 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f859ba936c9716dff56282f82e5671bbc5238ccd, data reload: false
------ Round 1 ----------------------------------
q1	17614	5380	5276	5276
q2	1930	281	187	187
q3	10299	1248	672	672
q4	10225	959	523	523
q5	7547	2243	2285	2243
q6	174	166	128	128
q7	875	774	604	604
q8	9285	1299	1072	1072
q9	6734	5023	5099	5023
q10	6880	2330	1917	1917
q11	447	266	255	255
q12	337	351	227	227
q13	17787	3437	2959	2959
q14	230	232	212	212
q15	526	473	463	463
q16	413	421	384	384
q17	539	813	341	341
q18	7515	6966	6969	6966
q19	1393	953	536	536
q20	317	304	216	216
q21	2767	2421	2338	2338
q22	1053	1050	970	970
Total cold run time: 104887 ms
Total hot run time: 33512 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5320	5258	5255	5255
q2	237	307	216	216
q3	2075	2530	2179	2179
q4	1290	1682	1341	1341
q5	4119	4008	4369	4008
q6	213	179	140	140
q7	1982	1925	1814	1814
q8	2556	2511	2407	2407
q9	7311	7147	7226	7147
q10	3178	3412	2927	2927
q11	539	520	504	504
q12	765	789	638	638
q13	3513	3691	3260	3260
q14	298	314	299	299
q15	479	447	443	443
q16	625	675	454	454
q17	1181	1583	1397	1397
q18	12674	7713	7547	7547
q19	2732	960	1049	960
q20	2065	1997	1802	1802
q21	10135	4320	4196	4196
q22	1041	1010	966	966
Total cold run time: 64328 ms
Total hot run time: 49900 ms

doris-robot · 2025-08-05T02:52:21Z

TPC-DS: Total hot run time: 160552 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f859ba936c9716dff56282f82e5671bbc5238ccd, data reload: false
reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL ============================================
query1	989	401	433	401
query2	6543	1773	1675	1675
query3	6749	231	218	218
query4	26859	23659	23213	23213
query5	4337	647	511	511
query6	322	256	211	211
query7	4638	520	286	286
query8	279	231	230	230
query9	8543	2897	2961	2897
query10	487	349	284	284
query11	15728	15192	15072	15072
query12	190	140	134	134
query13	1687	590	449	449
query14	9813	5814	5949	5814
query15	281	197	161	161
query16	7934	625	450	450
query17	1646	780	647	647
query18	2052	448	329	329
query19	309	217	194	194
query20	153	146	143	143
query21	217	127	112	112
query22	3946	4027	3746	3746
query23	34281	34280	34251	34251
query24	7550	2432	2419	2419
query25	533	510	454	454
query26	721	294	160	160
query27	2484	512	344	344
query28	3078	2316	2302	2302
query29	725	632	485	485
query30	297	232	207	207
query31	875	805	693	693
query32	86	73	85	73
query33	535	423	397	397
query34	785	868	518	518
query35	794	836	741	741
query36	1009	1035	959	959
query37	133	109	86	86
query38	3905	3958	3906	3906
query39	1450	1376	1390	1376
query40	237	143	135	135
query41	61	58	53	53
query42	136	124	124	124
query43	490	535	474	474
query44	1408	886	881	881
query45	202	188	181	181
query46	936	1063	671	671
query47	1798	1818	1729	1729
query48	391	421	342	342
query49	662	529	421	421
query50	654	693	412	412
query51	4155	4275	4098	4098
query52	127	130	120	120
query53	254	303	213	213
query54	664	649	567	567
query55	98	90	86	86
query56	353	355	343	343
query57	1219	1227	1136	1136
query58	335	323	331	323
query59	2611	2662	2532	2532
query60	414	393	383	383
query61	131	121	119	119
query62	794	761	677	677
query63	256	216	222	216
query64	2813	1069	784	784
query65	4184	4106	4084	4084
query66	885	440	342	342
query67	query68	18336	647	601	601
query69	1002	311	281	281
query70	1450	1122	1101	1101
query71	680	341	340	340
query72	9175	2253	4750	2253
query73	4743	772	364	364
query74	8988	9062	8702	8702
query75	7115	3116	2634	2634
query76	8449	1206	773	773
query77	1172	406	360	360
query78	query79	17371	642	594	594
query80	2992	576	507	507
query81	592	265	239	239
query82	465	150	126	126
query83	456	289	267	267
query84	313	94	85	85
query85	1814	381	419	381
query86	373	319	288	288
query87	4220	4119	4079	4079
query88	5750	2222	2263	2222
query89	529	356	316	316
query90	2624	241	241	241
query91	150	139	109	109
query92	87	71	67	67
query93	7094	982	664	664
query94	1177	391	286	286
query95	420	328	325	325
query96	502	587	281	281
query97	2681	2692	2580	2580
query98	248	221	227	221
query99	1554	1370	1271	1271
Total cold run time: 301733 ms
Total hot run time: 160552 ms

doris-robot · 2025-08-05T02:58:00Z

ClickBench: Total hot run time: 33.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f859ba936c9716dff56282f82e5671bbc5238ccd, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.60	0.11	0.11
query5	0.42	0.40	0.42
query6	1.18	0.65	0.64
query7	0.03	0.02	0.02
query8	0.06	0.04	0.03
query9	0.55	0.47	0.47
query10	0.51	0.53	0.53
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.64	0.63	0.63
query14	0.92	1.04	1.10
query15	0.98	0.88	0.92
query16	0.39	0.41	0.39
query17	1.07	1.07	1.04
query18	0.22	0.20	0.20
query19	1.89	1.93	1.81
query20	0.02	0.01	0.02
query21	15.38	0.87	0.57
query22	0.79	1.27	0.74
query23	14.75	1.18	0.67
query24	6.77	1.68	0.54
query25	0.51	0.20	0.10
query26	0.58	0.15	0.14
query27	0.07	0.05	0.06
query28	9.53	0.86	0.46
query29	12.69	3.83	3.31
query30	3.01	2.94	2.93
query31	2.81	0.56	0.39
query32	3.23	0.57	0.49
query33	3.02	3.23	3.24
query34	15.98	5.41	4.94
query35	4.90	5.10	5.05
query36	0.71	0.51	0.51
query37	0.10	0.08	0.08
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.4 s
Total hot run time: 33.04 s

englefly · 2025-08-07T02:32:41Z

run buildall

englefly · 2025-08-08T10:28:06Z

run buildall

doris-robot · 2025-08-08T11:00:44Z

TPC-H: Total hot run time: 33514 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 527f3e1f9b1bdc1bc3da8ff1045df4aa31f89432, data reload: false
------ Round 1 ----------------------------------
q1	17617	5317	5327	5317
q2	1944	261	180	180
q3	10324	1281	690	690
q4	10244	980	499	499
q5	7455	2347	2226	2226
q6	187	164	134	134
q7	905	762	615	615
q8	9316	1294	1051	1051
q9	6661	5076	5041	5041
q10	6886	2360	1972	1972
q11	457	280	253	253
q12	351	356	218	218
q13	17758	3415	2952	2952
q14	228	246	209	209
q15	514	484	467	467
q16	414	418	374	374
q17	567	819	344	344
q18	7366	6952	7012	6952
q19	1349	979	556	556
q20	323	305	215	215
q21	3368	3042	2257	2257
q22	1051	1038	992	992
Total cold run time: 105285 ms
Total hot run time: 33514 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5319	5349	5312	5312
q2	230	314	220	220
q3	2119	2532	2157	2157
q4	1272	1672	1284	1284
q5	4071	4225	4277	4225
q6	238	185	142	142
q7	1935	1936	1773	1773
q8	2535	2632	2429	2429
q9	7103	7071	7527	7071
q10	3179	3319	2989	2989
q11	558	506	516	506
q12	902	824	652	652
q13	3479	3669	3262	3262
q14	303	317	288	288
q15	506	452	441	441
q16	455	485	436	436
q17	1159	1474	1337	1337
q18	8051	7809	7651	7651
q19	12304	909	892	892
q20	1997	2017	1768	1768
q21	14852	4232	4272	4232
q22	1041	1034	989	989
Total cold run time: 73608 ms
Total hot run time: 50056 ms

doris-robot · 2025-08-08T11:12:25Z

TPC-DS: Total hot run time: 169861 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 527f3e1f9b1bdc1bc3da8ff1045df4aa31f89432, data reload: false
============================================
query1	983	406	419	406
query2	6539	1802	1672	1672
query3	6744	236	220	220
query4	27204	23297	23220	23220
query5	4412	619	509	509
query6	325	229	219	219
query7	4637	509	296	296
query8	276	240	228	228
query9	8549	2923	2953	2923
query10	471	341	281	281
query11	15722	15030	14842	14842
query12	188	143	132	132
query13	1654	547	414	414
query14	8700	5871	5822	5822
query15	221	183	163	163
query16	7139	660	485	485
query17	1244	762	642	642
query18	2007	451	335	335
query19	221	234	182	182
query20	150	149	141	141
query21	217	130	109	109
query22	3890	4067	3791	3791
query23	34412	34415	34236	34236
query24	5630	2369	2437	2369
query25	484	530	438	438
query26	720	286	157	157
query27	2332	488	351	351
query28	3138	2314	2319	2314
query29	699	583	489	489
query30	293	228	197	197
query31	883	790	720	720
query32	89	80	75	75
query33	532	404	365	365
query34	797	840	513	513
query35	806	839	752	752
query36	1008	1037	927	927
query37	133	107	94	94
query38	3881	3935	3860	3860
query39	1456	1373	1359	1359
query40	234	144	133	133
query41	60	59	54	54
query42	139	120	127	120
query43	520	512	488	488
query44	1410	881	847	847
query45	197	188	186	186
query46	942	1055	667	667
query47	1795	1812	1751	1751
query48	407	428	305	305
query49	692	498	418	418
query50	667	685	411	411
query51	4206	4271	4194	4194
query52	125	129	113	113
query53	266	280	226	226
query54	667	643	559	559
query55	96	88	86	86
query56	352	368	353	353
query57	1183	1235	1122	1122
query58	333	334	326	326
query59	2637	2677	2622	2622
query60	395	415	382	382
query61	120	122	129	122
query62	763	730	675	675
query63	249	218	221	218
query64	2350	1085	767	767
query65	4242	4109	4131	4109
query66	974	452	328	328
query67	query68	16695	895	980	895
query69	1050	276	288	276
query70	1390	1076	1099	1076
query71	733	315	315	315
query72	9164	2347	2193	2193
query73	3451	674	366	366
query74	9514	9057	8917	8917
query75	7528	3173	2646	2646
query76	8851	1200	800	800
query77	1220	418	328	328
query78	9446	10211	9148	9148
query79	14854	594	590	590
query80	1822	535	495	495
query81	571	259	226	226
query82	473	153	120	120
query83	393	279	275	275
query84	308	102	82	82
query85	1044	370	339	339
query86	357	325	297	297
query87	4185	4141	4060	4060
query88	5501	2252	2229	2229
query89	490	356	313	313
query90	2561	227	226	226
query91	143	140	108	108
query92	93	75	66	66
query93	6254	961	648	648
query94	1100	387	274	274
query95	402	331	315	315
query96	506	585	280	280
query97	2654	2733	2609	2609
query98	252	244	227	227
query99	1455	1423	1292	1292
Total cold run time: 299267 ms
Total hot run time: 169861 ms

doris-robot · 2025-08-08T11:17:53Z

ClickBench: Total hot run time: 32.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 527f3e1f9b1bdc1bc3da8ff1045df4aa31f89432, data reload: false
query1	0.05	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.39	0.42	0.46
query6	1.17	0.68	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.48	0.46
query10	0.51	0.52	0.52
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.65	0.67	0.64
query14	0.91	1.17	1.04
query15	0.90	0.92	0.89
query16	0.37	0.39	0.38
query17	1.05	1.05	1.08
query18	0.22	0.20	0.21
query19	1.98	1.78	1.84
query20	0.01	0.01	0.01
query21	15.41	0.87	0.57
query22	0.76	1.19	0.63
query23	14.99	1.20	0.61
query24	6.77	1.00	0.59
query25	0.50	0.23	0.14
query26	0.65	0.16	0.12
query27	0.07	0.05	0.06
query28	9.87	0.88	0.44
query29	12.60	3.87	3.35
query30	3.06	2.96	2.95
query31	2.81	0.56	0.40
query32	3.26	0.56	0.48
query33	2.95	3.17	3.28
query34	16.16	5.33	5.01
query35	4.87	4.96	5.03
query36	0.69	0.52	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 107.05 s
Total hot run time: 32.89 s

englefly · 2025-08-11T04:32:49Z

run buildall

doris-robot · 2025-08-11T05:04:42Z

TPC-H: Total hot run time: 33698 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d0af81ad287d76b9bedadeb1f6b32ea8eaf7592, data reload: false
------ Round 1 ----------------------------------
q1	17612	5361	5231	5231
q2	1899	281	156	156
q3	10355	1424	727	727
q4	10236	1027	523	523
q5	7385	1152	801	801
q6	173	162	136	136
q7	682	763	492	492
q8	9439	1466	1052	1052
q9	6835	5093	4998	4998
q10	6913	2345	1979	1979
q11	506	306	284	284
q12	363	372	243	243
q13	17780	3430	2999	2999
q14	243	261	238	238
q15	520	459	469	459
q16	423	416	384	384
q17	564	832	347	347
q18	7429	7015	6975	6975
q19	1340	994	557	557
q20	342	343	219	219
q21	4458	3911	3977	3911
q22	1091	1032	987	987
Total cold run time: 106588 ms
Total hot run time: 33698 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5473	5407	5530	5407
q2	251	319	227	227
q3	2104	2573	2190	2190
q4	1407	1729	1325	1325
q5	4294	4641	4751	4641
q6	224	185	139	139
q7	6885	6758	6559	6559
q8	2583	2565	2500	2500
q9	7019	7286	7292	7286
q10	3164	3365	2937	2937
q11	1333	1193	1183	1183
q12	928	974	605	605
q13	3430	3555	3268	3268
q14	322	318	278	278
q15	520	443	441	441
q16	445	461	442	442
q17	1255	1682	1370	1370
q18	7743	7793	7970	7793
q19	12203	987	924	924
q20	2860	1987	1890	1890
q21	16006	8330	8602	8330
q22	1129	1066	1049	1049
Total cold run time: 81578 ms
Total hot run time: 60784 ms

englefly · 2025-08-11T06:09:17Z

run buildall

doris-robot · 2025-08-11T06:34:35Z

TPC-H: Total hot run time: 34138 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 784f0d359ba7ebd730777f3239fd5897eccaff64, data reload: false
------ Round 1 ----------------------------------
q1	17603	5400	5328	5328
q2	1904	269	154	154
q3	10348	1521	723	723
q4	10238	1103	511	511
q5	7398	1250	830	830
q6	179	176	137	137
q7	755	741	471	471
q8	9431	1560	1209	1209
q9	6875	5211	5311	5211
q10	6969	2382	1959	1959
q11	509	298	277	277
q12	365	404	233	233
q13	17789	3550	2964	2964
q14	250	238	222	222
q15	540	461	461	461
q16	426	431	373	373
q17	559	852	363	363
q18	7663	7165	7065	7065
q19	1042	1076	590	590
q20	348	355	221	221
q21	4647	3852	4023	3852
q22	1076	1054	984	984
Total cold run time: 106914 ms
Total hot run time: 34138 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5663	5728	5736	5728
q2	254	329	235	235
q3	2125	2630	2195	2195
q4	1373	1786	1306	1306
q5	4586	4666	4647	4647
q6	269	193	150	150
q7	6836	6714	6526	6526
q8	2804	2975	2699	2699
q9	7099	7025	7919	7025
q10	3310	3350	2916	2916
q11	1314	1276	1177	1177
q12	705	735	718	718
q13	3565	3741	3291	3291
q14	300	322	288	288
q15	546	492	463	463
q16	490	479	449	449
q17	1207	1735	1420	1420
q18	9277	7923	7763	7763
q19	7294	1066	1335	1066
q20	2132	2088	1907	1907
q21	12717	8265	8702	8265
q22	1087	1079	1001	1001
Total cold run time: 74953 ms
Total hot run time: 61235 ms

englefly · 2025-08-13T11:59:23Z

run buildall

doris-robot · 2025-08-13T12:22:59Z

TPC-H: Total hot run time: 33817 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43403e3d1d76871d32a9bff57de449a4a3c2cac8, data reload: false
------ Round 1 ----------------------------------
q1	17602	5315	5094	5094
q2	1925	283	179	179
q3	10326	1270	713	713
q4	10210	1005	501	501
q5	7539	2382	2326	2326
q6	178	160	131	131
q7	897	746	584	584
q8	9296	1278	1057	1057
q9	6958	5187	5270	5187
q10	6882	2390	1978	1978
q11	490	290	287	287
q12	336	346	217	217
q13	17764	3611	3022	3022
q14	230	233	225	225
q15	540	488	489	488
q16	419	446	362	362
q17	620	859	352	352
q18	7494	6965	7054	6965
q19	1220	957	583	583
q20	341	339	217	217
q21	3821	3218	2338	2338
q22	1052	1028	1011	1011
Total cold run time: 106140 ms
Total hot run time: 33817 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5172	5176	5153	5153
q2	243	322	217	217
q3	2187	2670	2256	2256
q4	1353	1782	1323	1323
q5	4194	4430	4465	4430
q6	224	170	129	129
q7	2092	1941	1800	1800
q8	2639	2481	2662	2481
q9	7285	7321	7346	7321
q10	3142	3338	2809	2809
q11	571	536	492	492
q12	677	777	628	628
q13	3488	3905	3270	3270
q14	303	370	408	370
q15	505	471	478	471
q16	433	486	429	429
q17	1148	1550	1346	1346
q18	7988	7859	7556	7556
q19	838	858	956	858
q20	2053	2168	1881	1881
q21	4894	4471	4305	4305
q22	1086	1063	1000	1000
Total cold run time: 52515 ms
Total hot run time: 50525 ms

doris-robot · 2025-08-13T12:34:37Z

TPC-DS: Total hot run time: 184905 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43403e3d1d76871d32a9bff57de449a4a3c2cac8, data reload: false
query1	981	379	416	379
query2	6526	1750	1730	1730
query3	6741	221	218	218
query4	26603	23335	23135	23135
query5	4463	638	483	483
query6	331	219	199	199
query7	4644	507	296	296
query8	274	244	226	226
query9	8563	2849	2845	2845
query10	493	325	275	275
query11	15900	15449	14735	14735
query12	164	111	110	110
query13	1670	544	415	415
query14	8959	5718	5616	5616
query15	203	196	171	171
query16	7367	654	464	464
query17	1195	723	586	586
query18	1982	409	317	317
query19	193	189	180	180
query20	126	119	118	118
query21	210	123	106	106
query22	4221	4178	4099	4099
query23	34165	33291	33208	33208
query24	8134	2334	2330	2330
query25	551	481	394	394
query26	1233	267	156	156
query27	2760	494	352	352
query28	4324	2220	2242	2220
query29	777	557	450	450
query30	280	218	188	188
query31	886	773	714	714
query32	82	76	76	76
query33	563	390	338	338
query34	781	818	516	516
query35	790	802	765	765
query36	973	1008	912	912
query37	115	105	89	89
query38	4057	4116	4024	4024
query39	1493	1398	1438	1398
query40	212	121	117	117
query41	59	56	54	54
query42	124	109	109	109
query43	491	491	455	455
query44	1345	846	839	839
query45	174	172	157	157
query46	858	992	627	627
query47	1774	1810	1746	1746
query48	384	406	318	318
query49	713	499	389	389
query50	657	688	418	418
query51	4084	4188	4095	4095
query52	115	111	98	98
query53	226	261	192	192
query54	588	580	522	522
query55	91	85	88	85
query56	321	293	289	289
query57	1206	1206	1153	1153
query58	268	261	254	254
query59	2644	2719	2672	2672
query60	348	333	312	312
query61	121	118	123	118
query62	774	722	668	668
query63	221	187	183	183
query64	4316	1020	673	673
query65	4319	4164	4248	4164
query66	1161	410	317	317
query67	15414	15402	15341	15341
query68	8507	911	563	563
query69	468	335	290	290
query70	1267	1130	1142	1130
query71	458	320	298	298
query72	5603	4805	4953	4805
query73	777	708	346	346
query74	9301	9090	8847	8847
query75	3819	3047	2634	2634
query76	3690	1143	735	735
query77	803	415	328	328
query78	9543	9515	8824	8824
query79	4311	816	566	566
query80	670	536	470	470
query81	467	253	221	221
query82	658	135	109	109
query83	289	246	231	231
query84	293	107	83	83
query85	792	382	438	382
query86	345	314	278	278
query87	4212	4260	4114	4114
query88	2829	2175	2197	2175
query89	452	311	279	279
query90	1935	219	220	219
query91	143	147	114	114
query92	92	71	65	65
query93	2455	976	633	633
query94	689	383	301	301
query95	469	312	311	311
query96	485	589	275	275
query97	2632	2653	2570	2570
query98	230	210	208	208
query99	1485	1445	1303	1303
Total cold run time: 276950 ms
Total hot run time: 184905 ms

doris-robot · 2025-08-13T12:40:06Z

ClickBench: Total hot run time: 32.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43403e3d1d76871d32a9bff57de449a4a3c2cac8, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.41	0.43	0.41
query6	1.18	0.63	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.60	0.52	0.51
query10	0.58	0.56	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.63	0.62	0.61
query14	0.80	0.83	0.83
query15	0.88	0.86	0.87
query16	0.38	0.39	0.38
query17	1.06	1.08	1.04
query18	0.22	0.19	0.20
query19	1.94	1.81	1.87
query20	0.01	0.00	0.01
query21	15.39	0.95	0.56
query22	0.73	1.22	0.77
query23	14.84	1.35	0.62
query24	6.63	1.24	0.42
query25	0.50	0.10	0.16
query26	0.67	0.15	0.14
query27	0.06	0.06	0.05
query28	9.65	0.93	0.44
query29	12.65	3.96	3.27
query30	3.06	3.04	2.96
query31	2.82	0.59	0.38
query32	3.24	0.56	0.47
query33	3.12	3.14	3.12
query34	16.08	5.45	4.84
query35	4.90	4.85	4.95
query36	0.69	0.50	0.50
query37	0.09	0.08	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.19	0.15	0.13
query41	0.08	0.04	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.04
Total cold run time: 106.59 s
Total hot run time: 32.18 s

hello-stephen · 2025-08-13T13:02:45Z

FE UT Coverage Report

Increment line coverage 59.65% (34/57) 🎉
Increment coverage report
Complete coverage report

englefly · 2025-08-14T01:40:04Z

run external

feiniaofeiafei · 2025-08-14T03:28:28Z

+ DistributeHint hint = new DistributeHint(DistributeType.SHUFFLE_RIGHT,
+ new JoinSkewInfo(skewExpr, hotValues, false));
+ join.setHint(hint);
+ return SaltJoin.transform(join);


The current position of the saltJoin rule in the rewriter is placed after eliminate join. This is to prevent cases where some joins that could originally be eliminated (e.g., EliminateJoinByUnique) might no longer be eliminable after being rewritten by saltJoin. The code calling SaltJoin here could cause this problem. Is it possible to add the hint directly without applying SaltJoin.transform(join)? Then, when a salt join is encountered later and the pattern matches, the saltJoin rule will automatically be applied?

this rule also need to be applies after EliminateJoinByUnique.

feiniaofeiafei · 2025-08-14T03:31:34Z

+ * add salt
+ */
+ public static Plan transform(LogicalJoin<Plan, Plan> join) {
 DistributeHint hint = join.getDistributeHint();


maybe we should check ConnectContext.get() != null

englefly · 2025-08-15T08:06:10Z

run buildall

doris-robot · 2025-08-15T08:37:19Z

TPC-H: Total hot run time: 33775 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 165bf59aae50dd09ce26d78e5476a7d421ae8420, data reload: false
------ Round 1 ----------------------------------
q1	17613	5145	5064	5064
q2	1909	271	182	182
q3	10329	1274	698	698
q4	10211	987	505	505
q5	7523	2302	2323	2302
q6	174	158	131	131
q7	889	746	597	597
q8	9287	1333	1074	1074
q9	6805	5158	5115	5115
q10	6897	2350	1989	1989
q11	481	278	271	271
q12	346	352	218	218
q13	17759	3588	2982	2982
q14	226	246	221	221
q15	566	480	478	478
q16	408	418	362	362
q17	578	851	360	360
q18	7365	7128	7286	7128
q19	1072	941	563	563
q20	345	316	214	214
q21	3787	2528	2319	2319
q22	1082	1039	1002	1002
Total cold run time: 105652 ms
Total hot run time: 33775 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5149	5089	5145	5089
q2	244	319	221	221
q3	2195	2631	2290	2290
q4	1330	1748	1354	1354
q5	4180	4275	4457	4275
q6	222	170	138	138
q7	2030	2011	1811	1811
q8	2611	2720	2604	2604
q9	7339	7259	7344	7259
q10	3107	3289	2835	2835
q11	599	517	487	487
q12	705	809	635	635
q13	3554	3917	3315	3315
q14	291	316	275	275
q15	512	474	468	468
q16	450	487	461	461
q17	1174	1595	1390	1390
q18	7879	7673	7431	7431
q19	789	767	791	767
q20	1896	2020	1789	1789
q21	4755	4367	4322	4322
q22	1085	1045	1021	1021
Total cold run time: 52096 ms
Total hot run time: 50237 ms

doris-robot · 2025-08-15T08:48:47Z

TPC-DS: Total hot run time: 184165 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 165bf59aae50dd09ce26d78e5476a7d421ae8420, data reload: false
query1	983	375	400	375
query2	6522	1772	1789	1772
query3	6769	223	220	220
query4	26299	23337	23160	23160
query5	4352	603	474	474
query6	313	221	212	212
query7	4628	488	288	288
query8	287	242	220	220
query9	8581	2875	2888	2875
query10	462	337	276	276
query11	15787	14915	14722	14722
query12	163	110	113	110
query13	1644	539	404	404
query14	8875	5638	5714	5638
query15	195	181	162	162
query16	7267	641	490	490
query17	1302	713	585	585
query18	1975	407	316	316
query19	228	184	162	162
query20	128	124	120	120
query21	215	129	123	123
query22	4410	4313	4084	4084
query23	34145	33167	33317	33167
query24	8180	2330	2349	2330
query25	528	463	415	415
query26	1237	275	156	156
query27	2756	498	337	337
query28	4499	2251	2198	2198
query29	773	559	443	443
query30	280	219	188	188
query31	863	803	723	723
query32	87	77	72	72
query33	577	391	370	370
query34	779	838	497	497
query35	791	817	737	737
query36	984	1029	919	919
query37	117	102	95	95
query38	4034	3984	4024	3984
query39	1513	1400	1461	1400
query40	208	124	115	115
query41	57	57	52	52
query42	114	104	107	104
query43	498	495	455	455
query44	1295	849	837	837
query45	171	161	164	161
query46	853	1002	634	634
query47	1775	1820	1763	1763
query48	372	408	313	313
query49	744	504	382	382
query50	633	676	396	396
query51	4092	4104	4019	4019
query52	121	112	105	105
query53	224	255	186	186
query54	587	587	510	510
query55	89	83	81	81
query56	310	294	293	293
query57	1193	1180	1133	1133
query58	281	262	275	262
query59	2698	2698	2608	2608
query60	338	327	302	302
query61	124	115	121	115
query62	804	705	653	653
query63	224	185	185	185
query64	4323	988	674	674
query65	4295	4208	4215	4208
query66	1148	413	316	316
query67	15602	15295	15008	15008
query68	6187	912	567	567
query69	511	332	295	295
query70	1190	1102	1156	1102
query71	411	324	306	306
query72	5357	4842	4755	4755
query73	626	570	344	344
query74	8996	9041	8655	8655
query75	3134	3059	2655	2655
query76	3179	1121	730	730
query77	492	389	310	310
query78	9539	9633	8899	8899
query79	2618	820	598	598
query80	744	532	455	455
query81	493	257	221	221
query82	449	126	104	104
query83	255	238	230	230
query84	244	104	87	87
query85	802	373	336	336
query86	387	298	303	298
query87	4260	4324	4158	4158
query88	3640	2180	2154	2154
query89	381	313	280	280
query90	1915	218	210	210
query91	141	154	107	107
query92	87	70	65	65
query93	2257	1001	630	630
query94	712	403	306	306
query95	384	313	298	298
query96	476	570	273	273
query97	2649	2709	2546	2546
query98	234	231	219	219
query99	1345	1396	1283	1283
Total cold run time: 270796 ms
Total hot run time: 184165 ms

doris-robot · 2025-08-15T08:54:16Z

ClickBench: Total hot run time: 32.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 165bf59aae50dd09ce26d78e5476a7d421ae8420, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.07	0.08
query4	1.63	0.10	0.12
query5	0.43	0.43	0.42
query6	1.17	0.62	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.60	0.52	0.52
query10	0.57	0.57	0.57
query11	0.16	0.11	0.11
query12	0.16	0.11	0.11
query13	0.67	0.61	0.61
query14	0.80	0.83	0.81
query15	0.84	0.85	0.84
query16	0.38	0.38	0.38
query17	1.03	1.02	1.07
query18	0.21	0.20	0.20
query19	1.86	1.84	1.84
query20	0.01	0.00	0.01
query21	15.40	0.93	0.55
query22	0.79	1.09	0.68
query23	15.00	1.37	0.63
query24	6.56	1.01	1.59
query25	0.50	0.25	0.07
query26	0.44	0.16	0.13
query27	0.06	0.06	0.06
query28	10.25	0.95	0.44
query29	12.61	3.95	3.25
query30	3.15	2.97	3.03
query31	2.83	0.57	0.38
query32	3.23	0.55	0.47
query33	3.04	3.07	3.08
query34	16.08	5.44	4.86
query35	4.89	4.90	5.01
query36	0.68	0.52	0.49
query37	0.09	0.08	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.96 s
Total hot run time: 32.65 s

englefly · 2025-08-19T00:26:01Z

run cloud_p0

github-actions · 2025-08-28T03:45:04Z

PR approved by anyone and no changes requested.

github-actions · 2025-08-29T07:20:37Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? Related PR: #59591 #54207 Problem Summary: This PR restricts the auto salt join rewrite in two cases. First, ASOF joins are excluded from automatic salt join optimization because the current rewrite may introduce large build-side expansion and is not cost guarded for ASOF semantics. Second, SaltJoin now marks internally generated helper joins with `isSaltJoinGenerated` in `JoinReorderContext`. `SkewJoin` skips these generated joins to avoid applying salt rewrite recursively on SaltJoin's own expansion plan.

englefly changed the title ~~Auto salt join~~[feat](nereids)using salt-join automatically when encountering skew joinAug 5, 2025

englefly force-pushed the auto-salt-join branch from f859ba9 to af92114Compare August 7, 2025 02:32

englefly force-pushed the auto-salt-join branch from 527f3e1 to 8d0af81Compare August 11, 2025 04:32

englefly force-pushed the auto-salt-join branch from 8d0af81 to 784f0d3Compare August 11, 2025 06:09

engleflyand others added 2 commits August 13, 2025 17:15

skew-join
8c91e67

fix0813
43403e3

englefly force-pushed the auto-salt-join branch from 784f0d3 to 43403e3Compare August 13, 2025 09:32

feiniaofeiafei reviewed Aug 14, 2025
View reviewed changes

rule order
165bf59

englefly marked this pull request as ready for review August 15, 2025 08:06

feiniaofeiafei approved these changes Aug 28, 2025
View reviewed changes

github-actionsBot added the reviewed label Aug 28, 2025

starocean999 approved these changes Aug 29, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2025

englefly merged commit 1faf573 into apache:masterAug 29, 2025
27 of 28 checks passed

englefly deleted the auto-salt-join branch August 29, 2025 07:21

feiniaofeiafei mentioned this pull request Jun 16, 2026
[fix](join) Restrict auto salt join rewrite #64518
Merged

Uh oh!

Conversation

englefly commented Aug 1, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Aug 1, 2025

Uh oh!

englefly commented Aug 5, 2025

Uh oh!

doris-robot commented Aug 5, 2025

Uh oh!

doris-robot commented Aug 5, 2025

Uh oh!

doris-robot commented Aug 5, 2025

Uh oh!

englefly commented Aug 7, 2025

Uh oh!

englefly commented Aug 8, 2025

Uh oh!

doris-robot commented Aug 8, 2025

Uh oh!

doris-robot commented Aug 8, 2025

Uh oh!

doris-robot commented Aug 8, 2025

Uh oh!

englefly commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

englefly commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

englefly commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

hello-stephen commented Aug 13, 2025

FE UT Coverage Report

Uh oh!

englefly commented Aug 14, 2025

Uh oh!

feiniaofeiafeiAug 14, 2025

Choose a reason for hiding this comment

Uh oh!

engleflyAug 14, 2025

Choose a reason for hiding this comment

Uh oh!

feiniaofeiafeiAug 14, 2025

Choose a reason for hiding this comment

Uh oh!

englefly commented Aug 15, 2025

Uh oh!

doris-robot commented Aug 15, 2025

Uh oh!

doris-robot commented Aug 15, 2025

Uh oh!

doris-robot commented Aug 15, 2025

Uh oh!

englefly commented Aug 19, 2025

Uh oh!

github-actionsBot commented Aug 28, 2025

Uh oh!

github-actionsBot commented Aug 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

englefly commented Aug 1, 2025•
edited
Loading