[feature](nereids)support mark join by starocean999 · Pull Request #30133 · apache/doris

starocean999 · 2024-01-19T07:42:15Z

Proposed changes

support mark join
be part is implement by Author: Jerry Hu <mrhhsg@gmail.com>

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

starocean999 · 2024-01-19T07:42:35Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-01-19T07:53:25Z

+// specific language governing permissions and limitations
+// under the License.
+
+#include <gen_cpp/PlanNodes_types.h>


warning: 'gen_cpp/PlanNodes_types.h' file not found [clang-diagnostic-error]
#include<gen_cpp/PlanNodes_types.h> ^

github-actions · 2024-01-19T07:53:25Z

 std::vector<uint8_t>& visited, bool has_null_in_build_side);

+ template <bool with_other_conjuncts>
+ Status do_mark_join_conjuncts(Block* output_block, const size_t hash_table_bucket_size);


warning: parameter 'hash_table_bucket_size' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
Suggested change
Statusdo_mark_join_conjuncts(Block*output_block, constsize_thash_table_bucket_size);
Statusdo_mark_join_conjuncts(Block*output_block, size_thash_table_bucket_size);

github-actions · 2024-01-19T07:53:25Z


+template <int JoinOpType, typename Parent>
+template <bool with_other_conjuncts>
+Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts(


warning: function 'do_mark_join_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^
Additional context
be/src/vec/exec/join/process_hash_table_probe_impl.h:228: 85 lines including whitespace and comments (threshold 80)
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^

github-actions · 2024-01-19T07:53:25Z

 return Status::OK();
 }

+Status VExprContext::execute_conjuncts(const VExprContextSPtrs& conjuncts, Block* block,


warning: method 'execute_conjuncts' can be made static [readability-convert-member-functions-to-static]
Suggested change
Status VExprContext::execute_conjuncts(const VExprContextSPtrs& conjuncts, Block* block,
staticStatus VExprContext::execute_conjuncts(const VExprContextSPtrs& conjuncts, Block* block,

doris-robot · 2024-01-19T09:09:33Z

TeamCity be ut coverage result:
Function Coverage: 36.72% (8653/23568)
Line Coverage: 28.74% (70664/245889)
Region Coverage: 27.62% (36442/131935)
Branch Coverage: 24.35% (18650/76602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8d1b65c47efdb597089c9404a9571e743de2da6f_8d1b65c47efdb597089c9404a9571e743de2da6f/report/index.html

starocean999 · 2024-01-19T10:19:49Z

run buildall

doris-robot · 2024-01-19T10:52:49Z

TeamCity be ut coverage result:
Function Coverage: 36.72% (8655/23568)
Line Coverage: 28.75% (70687/245894)
Region Coverage: 27.63% (36457/131936)
Branch Coverage: 24.36% (18657/76604)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c072fdcb2bd1be36af977edfc6f4a3a4d00db135_c072fdcb2bd1be36af977edfc6f4a3a4d00db135/report/index.html

starocean999 · 2024-01-20T01:41:03Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-01-20T01:47:04Z


+template <int JoinOpType, typename Parent>
+template <bool with_other_conjuncts>
+Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts(


warning: function 'do_mark_join_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^
Additional context
be/src/vec/exec/join/process_hash_table_probe_impl.h:229: 82 lines including whitespace and comments (threshold 80)
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^

mrhhsg · 2024-01-20T14:03:57Z

run buildall

doris-robot · 2024-01-20T14:33:10Z

TeamCity be ut coverage result:
Function Coverage: 36.71% (8652/23568)
Line Coverage: 28.74% (70664/245915)
Region Coverage: 27.62% (36442/131956)
Branch Coverage: 24.34% (18653/76620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5b4eb16b84525f8cf77032df37352cf90340bdd3_5b4eb16b84525f8cf77032df37352cf90340bdd3/report/index.html

starocean999 · 2024-01-21T01:52:21Z

run buildall

doris-robot · 2024-01-21T02:23:11Z

TeamCity be ut coverage result:
Function Coverage: 36.72% (8655/23568)
Line Coverage: 28.74% (70679/245915)
Region Coverage: 27.63% (36453/131956)
Branch Coverage: 24.35% (18654/76620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bba8828c93c94d9858efa01751754c666859b914_bba8828c93c94d9858efa01751754c666859b914/report/index.html

doris-robot · 2024-01-21T02:24:56Z

TPC-H: Total hot run time: 38586 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bba8828c93c94d9858efa01751754c666859b914, data reload: false
------ Round 1 ----------------------------------
q1	17615	5321	5218	5218
q2	2035	134	129	129
q3	10642	1131	1128	1128
q4	10224	796	836	796
q5	7732	3116	3154	3116
q6	196	119	118	118
q7	842	494	488	488
q8	9241	1906	1957	1906
q9	7189	6357	6328	6328
q10	8211	3052	3052	3052
q11	417	209	208	208
q12	353	184	181	181
q13	17984	3350	3327	3327
q14	247	206	211	206
q15	549	513	502	502
q16	433	394	374	374
q17	925	502	459	459
q18	7247	6885	6639	6639
q19	1573	1338	1391	1338
q20	557	321	286	286
q21	2795	2491	2490	2490
q22	364	297	315	297
Total cold run time: 107371 ms
Total hot run time: 38586 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5339	5210	5160	5160
q2	334	216	210	210
q3	3329	3226	3194	3194
q4	2044	2031	2066	2031
q5	5915	5806	5962	5806
q6	198	117	115	115
q7	2299	1816	1902	1816
q8	3225	3351	3377	3351
q9	8910	8781	8758	8758
q10	3948	3813	3770	3770
q11	552	437	449	437
q12	809	588	670	588
q13	16936	3164	3111	3111
q14	277	264	266	264
q15	550	503	510	503
q16	524	443	462	443
q17	1837	1810	1836	1810
q18	9570	16738	9702	9702
q19	25913	1538	1543	1538
q20	4576	1913	1936	1913
q21	14834	5610	5771	5610
q22	981	517	522	517
Total cold run time: 112900 ms
Total hot run time: 60647 ms

doris-robot · 2024-01-21T02:36:14Z

TPC-DS: Total hot run time: 176927 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bba8828c93c94d9858efa01751754c666859b914, data reload: false
query1	918	339	329	329
query2	6568	1941	1987	1941
query3	6695	205	203	203
query4	31987	22283	22144	22144
query5	6895	541	598	541
query6	254	160	154	154
query7	4608	265	256	256
query8	221	171	187	171
query9	8333	2532	2541	2532
query10	423	211	212	211
query11	16578	15567	15415	15415
query12	117	69	67	67
query13	1677	387	386	386
query14	10653	7119	7012	7012
query15	219	177	181	177
query16	4597	237	231	231
query17	989	480	468	468
query18	1783	262	255	255
query19	174	141	137	137
query20	71	67	65	65
query21	189	136	135	135
query22	5014	4832	4538	4538
query23	31742	30931	30987	30931
query24	7672	2846	2821	2821
query25	406	327	307	307
query26	726	144	143	143
query27	2311	269	274	269
query28	3966	1833	1829	1829
query29	921	623	647	623
query30	201	138	143	138
query31	916	734	764	734
query32	66	48	48	48
query33	420	217	209	209
query34	827	461	465	461
query35	882	789	798	789
query36	1291	1205	1227	1205
query37	92	60	62	60
query38	3331	3227	3234	3227
query39	1327	1271	1259	1259
query40	201	88	87	87
query41	39	35	34	34
query42	93	80	91	80
query43	511	490	515	490
query44	1037	680	680	680
query45	194	181	172	172
query46	1039	651	642	642
query47	1641	1533	1578	1533
query48	386	316	321	316
query49	619	286	287	286
query50	680	320	315	315
query51	5338	5204	5197	5197
query52	88	79	76	76
query53	321	261	254	254
query54	438	292	283	283
query55	85	76	74	74
query56	181	167	177	167
query57	1023	919	923	919
query58	185	163	156	156
query59	3077	2832	2672	2672
query60	204	181	184	181
query61	82	83	81	81
query62	449	360	363	360
query63	286	264	268	264
query64	2556	1794	1749	1749
query65	3343	3273	3268	3268
query66	1037	314	311	311
query67	15531	15058	15111	15058
query68	12515	499	512	499
query69	626	303	296	296
query70	1563	1481	1437	1437
query71	498	218	217	217
query72	5160	2868	2829	2829
query73	3033	316	315	315
query74	7236	6377	6451	6377
query75	5471	2299	2302	2299
query76	6486	1056	1017	1017
query77	677	244	237	237
query78	9841	8702	8740	8702
query79	1000	510	527	510
query80	609	319	322	319
query81	473	206	203	203
query82	206	86	78	78
query83	139	118	120	118
query84	283	68	69	68
query85	1089	338	321	321
query86	379	374	383	374
query87	3518	3317	3330	3317
query88	2757	2200	2195	2195
query89	433	343	348	343
query90	2225	186	185	185
query91	159	122	129	122
query92	54	41	42	41
query93	1534	426	407	407
query94	1449	157	155	155
query95	498	465	446	446
query96	625	318	316	316
query97	4282	4149	4153	4149
query98	199	195	186	186
query99	988	723	724	723
Total cold run time: 282729 ms
Total hot run time: 176927 ms

doris-robot · 2024-01-21T02:41:38Z

ClickBench: Total hot run time: 30.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bba8828c93c94d9858efa01751754c666859b914, data reload: false
query1	0.03	0.02	0.02
query2	0.05	0.02	0.02
query3	0.22	0.05	0.05
query4	1.71	0.06	0.08
query5	0.55	0.52	0.53
query6	1.30	0.64	0.62
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.55	0.49	0.48
query10	0.55	0.56	0.55
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.60	0.61	0.60
query14	0.79	0.83	0.79
query15	0.79	0.76	0.78
query16	0.38	0.37	0.37
query17	0.99	1.02	0.98
query18	0.24	0.25	0.26
query19	1.80	1.77	1.79
query20	0.01	0.01	0.01
query21	15.42	0.61	0.59
query22	2.49	2.04	1.92
query23	17.17	0.86	0.74
query24	2.40	1.28	0.37
query25	0.41	0.40	0.05
query26	0.39	0.14	0.13
query27	0.05	0.06	0.04
query28	11.96	0.76	0.77
query29	12.49	3.16	3.20
query30	0.55	0.51	0.47
query31	2.78	0.34	0.34
query32	3.39	0.47	0.50
query33	3.23	3.22	3.23
query34	15.79	4.32	4.27
query35	4.36	4.31	4.29
query36	1.10	1.08	1.08
query37	0.07	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.01
Total cold run time: 105.23 s
Total hot run time: 30.2 s

doris-robot · 2024-01-21T02:44:43Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bba8828c93c94d9858efa01751754c666859b914 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.7 seconds inserted 10000000 Rows, about 680K ops/s

mrhhsg · 2024-01-21T03:12:22Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-01-21T03:22:36Z


+template <int JoinOpType, typename Parent>
+template <bool with_other_conjuncts>
+Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts(


warning: function 'do_mark_join_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^
Additional context
be/src/vec/exec/join/process_hash_table_probe_impl.h:229: 89 lines including whitespace and comments (threshold 80)
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^

doris-robot · 2024-01-21T04:13:24Z

TPC-H: Total hot run time: 39128 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4988f8b78131587ac25a63c8f789915f77dd4b7, data reload: false
------ Round 1 ----------------------------------
q1	17661	5837	5198	5198
q2	2049	142	140	140
q3	10625	1170	1151	1151
q4	10233	790	810	790
q5	7756	3218	3183	3183
q6	200	122	121	121
q7	850	517	484	484
q8	9196	1944	1998	1944
q9	7273	6411	6389	6389
q10	8188	3061	3042	3042
q11	417	218	205	205
q12	362	193	196	193
q13	17990	3379	3408	3379
q14	251	228	221	221
q15	554	502	512	502
q16	448	378	405	378
q17	945	544	539	539
q18	7664	7083	6861	6861
q19	1590	1426	1341	1341
q20	605	278	317	278
q21	2828	2481	2527	2481
q22	347	318	308	308
Total cold run time: 108032 ms
Total hot run time: 39128 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5558	5311	5312	5311
q2	333	218	216	216
q3	3351	3196	3248	3196
q4	2131	2091	2072	2072
q5	6001	5962	5914	5914
q6	196	119	117	117
q7	2321	1913	1860	1860
q8	3233	3383	3434	3383
q9	9107	8862	8749	8749
q10	3951	3827	3842	3827
q11	567	462	456	456
q12	794	615	643	615
q13	16933	3209	3195	3195
q14	288	260	275	260
q15	560	512	507	507
q16	519	478	484	478
q17	1927	1895	1832	1832
q18	9695	17518	9588	9588
q19	24416	1508	1540	1508
q20	4618	1944	1914	1914
q21	14810	5697	5663	5663
q22	973	561	549	549
Total cold run time: 112282 ms
Total hot run time: 61210 ms

doris-robot · 2024-01-21T04:25:04Z

TPC-DS: Total hot run time: 177193 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4988f8b78131587ac25a63c8f789915f77dd4b7, data reload: false
query1	936	333	331	331
query2	6564	2004	2061	2004
query3	6691	204	201	201
query4	33155	22108	22110	22108
query5	6949	586	524	524
query6	244	159	156	156
query7	4597	266	257	257
query8	224	177	183	177
query9	8432	2531	2537	2531
query10	426	232	212	212
query11	17045	15649	15445	15445
query12	119	68	67	67
query13	1678	383	367	367
query14	10597	7072	6912	6912
query15	220	180	182	180
query16	4602	233	239	233
query17	987	476	473	473
query18	1788	260	253	253
query19	179	137	141	137
query20	69	68	71	68
query21	198	127	130	127
query22	4960	4756	4780	4756
query23	31813	30747	30881	30747
query24	12953	2864	2843	2843
query25	569	326	305	305
query26	1811	143	142	142
query27	3343	268	273	268
query28	7602	1815	1797	1797
query29	2152	636	628	628
query30	287	137	144	137
query31	929	742	743	742
query32	74	52	50	50
query33	704	218	217	217
query34	1268	463	456	456
query35	886	779	765	765
query36	1307	1206	1149	1149
query37	179	61	60	60
query38	3413	3232	3270	3232
query39	1357	1276	1257	1257
query40	339	89	88	88
query41	39	35	34	34
query42	90	86	84	84
query43	545	493	516	493
query44	1136	693	682	682
query45	191	179	178	178
query46	1074	641	643	641
query47	1659	1535	1601	1535
query48	387	306	308	306
query49	1215	294	282	282
query50	667	305	307	305
query51	5372	5186	5306	5186
query52	88	80	76	76
query53	322	261	267	261
query54	457	296	289	289
query55	80	77	75	75
query56	180	165	160	160
query57	1024	939	974	939
query58	185	169	171	169
query59	3030	2833	2829	2829
query60	207	186	199	186
query61	85	85	82	82
query62	643	359	350	350
query63	288	267	257	257
query64	6179	1778	1783	1778
query65	3360	3255	3259	3255
query66	1415	316	316	316
query67	15437	15386	15293	15293
query68	12712	513	498	498
query69	619	310	302	302
query70	1738	1435	1428	1428
query71	554	220	211	211
query72	4941	2828	2846	2828
query73	2958	313	310	310
query74	7376	6471	6428	6428
query75	5387	2281	2292	2281
query76	6743	1011	1044	1011
query77	890	242	232	232
query78	9705	8610	8734	8610
query79	1318	505	495	495
query80	603	330	320	320
query81	459	212	211	211
query82	205	90	84	84
query83	201	121	122	121
query84	275	70	74	70
query85	1121	351	342	342
query86	400	367	383	367
query87	3566	3288	3349	3288
query88	3178	2182	2196	2182
query89	434	371	364	364
query90	2103	187	184	184
query91	159	130	131	130
query92	57	47	45	45
query93	1386	420	428	420
query94	1299	156	159	156
query95	504	457	452	452
query96	612	319	318	318
query97	4295	4170	4173	4170
query98	203	191	183	183
query99	1112	709	709	709
Total cold run time: 304114 ms
Total hot run time: 177193 ms

doris-robot · 2024-01-21T04:30:30Z

ClickBench: Total hot run time: 31.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4988f8b78131587ac25a63c8f789915f77dd4b7, data reload: false
query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.23	0.07	0.06
query4	1.69	0.08	0.08
query5	0.53	0.52	0.53
query6	1.26	0.64	0.61
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.54	0.49	0.51
query10	0.57	0.56	0.56
query11	0.11	0.09	0.08
query12	0.11	0.09	0.08
query13	0.60	0.62	0.61
query14	0.78	0.81	0.80
query15	0.81	0.77	0.76
query16	0.36	0.36	0.38
query17	1.02	1.01	1.03
query18	0.22	0.27	0.25
query19	1.85	1.79	1.76
query20	0.02	0.01	0.01
query21	15.41	0.60	0.57
query22	2.33	2.23	2.05
query23	17.51	0.93	0.92
query24	2.34	1.58	1.24
query25	0.41	0.14	0.14
query26	0.68	0.14	0.14
query27	0.06	0.05	0.05
query28	10.98	0.77	0.76
query29	12.50	3.31	3.20
query30	0.57	0.50	0.47
query31	2.78	0.34	0.35
query32	3.36	0.48	0.48
query33	3.23	3.24	3.24
query34	15.72	4.32	4.30
query35	4.41	4.24	4.30
query36	1.13	1.07	1.06
query37	0.06	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.12	0.13
query41	0.07	0.02	0.01
query42	0.03	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.66 s
Total hot run time: 31.57 s

doris-robot · 2024-01-21T04:33:34Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d4988f8b78131587ac25a63c8f789915f77dd4b7 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.9 seconds inserted 10000000 Rows, about 671K ops/s

doris-robot · 2024-01-21T04:33:58Z

TeamCity be ut coverage result:
Function Coverage: 36.70% (8655/23583)
Line Coverage: 28.73% (70675/245978)
Region Coverage: 27.61% (36442/131991)
Branch Coverage: 24.34% (18655/76642)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d4988f8b78131587ac25a63c8f789915f77dd4b7_d4988f8b78131587ac25a63c8f789915f77dd4b7/report/index.html

github-actions

clang-tidy made some suggestions

github-actions · 2024-01-21T11:47:32Z


+template <int JoinOpType, typename Parent>
+template <bool with_other_conjuncts>
+Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts(


warning: function 'do_mark_join_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^
Additional context
be/src/vec/exec/join/process_hash_table_probe_impl.h:233: 82 lines including whitespace and comments (threshold 80)
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^

mrhhsg · 2024-01-21T12:22:13Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-01-21T12:31:25Z


+template <int JoinOpType, typename Parent>
+template <bool with_other_conjuncts>
+Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts(


warning: function 'do_mark_join_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^
Additional context
be/src/vec/exec/join/process_hash_table_probe_impl.h:233: 89 lines including whitespace and comments (threshold 80)
Status ProcessHashTableProbe<JoinOpType, Parent>::do_mark_join_conjuncts( ^

doris-robot · 2024-01-21T12:53:17Z

TeamCity be ut coverage result:
Function Coverage: 36.71% (8657/23583)
Line Coverage: 28.74% (70687/245978)
Region Coverage: 27.62% (36455/131991)
Branch Coverage: 24.33% (18649/76642)
Coverage Report: http://coverage.selectdb-in.cc/coverage/349f75e13bda39ef822053ebd2717a05c28d8137_349f75e13bda39ef822053ebd2717a05c28d8137/report/index.html

Co-authored-by: Jerry Hu <mrhhsg@gmail.com>

starocean999 · 2024-01-26T07:35:49Z

run buildall

doris-robot · 2024-01-26T08:28:18Z

TPC-H: Total hot run time: 39545 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66b6a556d542663e9086cf05c6c01aabb57fbc65, data reload: false
------ Round 1 ----------------------------------
q1	18990	5494	5284	5284
q2	2480	155	139	139
q3	11746	1208	1201	1201
q4	11008	829	837	829
q5	8163	3254	3234	3234
q6	208	123	126	123
q7	921	526	506	506
q8	9584	1990	1990	1990
q9	7362	6432	6379	6379
q10	8197	2980	3054	2980
q11	416	213	215	213
q12	365	201	197	197
q13	17991	3421	3403	3403
q14	245	214	215	214
q15	555	517	504	504
q16	434	371	383	371
q17	949	565	535	535
q18	7509	6969	6901	6901
q19	1589	1455	1420	1420
q20	608	302	299	299
q21	2881	2528	2532	2528
q22	365	319	295	295
Total cold run time: 112566 ms
Total hot run time: 39545 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5636	5247	5516	5247
q2	334	214	215	214
q3	3377	3259	3258	3258
q4	2123	2059	2097	2059
q5	6078	5969	5915	5915
q6	202	118	123	118
q7	2328	1907	1898	1898
q8	3229	3395	3406	3395
q9	9023	8838	8943	8838
q10	3988	3835	3797	3797
q11	585	458	445	445
q12	813	628	643	628
q13	16914	3186	3168	3168
q14	301	263	276	263
q15	575	511	505	505
q16	516	480	453	453
q17	1892	1859	1840	1840
q18	9520	9439	10087	9439
q19	19186	1578	1523	1523
q20	4602	1955	1924	1924
q21	14492	5640	5617	5617
q22	981	560	564	560
Total cold run time: 106695 ms
Total hot run time: 61104 ms

doris-robot · 2024-01-26T08:40:08Z

TPC-DS: Total hot run time: 185839 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66b6a556d542663e9086cf05c6c01aabb57fbc65, data reload: false
query1	943	334	337	334
query2	6543	1953	1929	1929
query3	6713	205	200	200
query4	31732	22265	22194	22194
query5	4463	440	369	369
query6	247	148	159	148
query7	4600	262	256	256
query8	245	184	172	172
query9	8474	2594	2581	2581
query10	411	217	209	209
query11	16752	15566	15577	15566
query12	112	72	65	65
query13	1672	380	379	379
query14	10645	6954	7004	6954
query15	216	184	184	184
query16	5793	262	254	254
query17	934	481	466	466
query18	1784	259	253	253
query19	173	131	129	129
query20	77	76	70	70
query21	187	133	134	133
query22	4949	4800	4424	4424
query23	31609	30808	30815	30808
query24	7705	2814	2778	2778
query25	437	334	310	310
query26	805	138	135	135
query27	2620	277	285	277
query28	3757	1850	1823	1823
query29	964	615	613	613
query30	204	137	137	137
query31	927	751	745	745
query32	83	49	51	49
query33	421	207	202	202
query34	816	441	459	441
query35	851	776	770	770
query36	1264	1221	1092	1092
query37	94	64	56	56
query38	3383	3206	3220	3206
query39	1327	1256	1267	1256
query40	192	87	82	82
query41	38	35	34	34
query42	87	81	83	81
query43	481	449	503	449
query44	1026	676	687	676
query45	193	173	174	173
query46	1052	641	636	636
query47	1625	1581	1528	1528
query48	393	299	321	299
query49	633	299	290	290
query50	680	306	311	306
query51	5360	5151	5211	5151
query52	98	73	81	73
query53	323	252	248	248
query54	219	180	194	180
query55	81	72	73	72
query56	183	154	156	154
query57	980	953	937	937
query58	177	159	163	159
query59	2938	2746	2739	2739
query60	204	176	172	172
query61	85	85	81	81
query62	437	348	367	348
query63	282	257	262	257
query64	3333	1770	1768	1768
query65	3344	3274	3241	3241
query66	1165	333	312	312
query67	15383	14774	14962	14774
query68	14891	489	491	489
query69	611	289	282	282
query70	2284	1521	1435	1435
query71	10451	10198	10196	10196
query72	5115	2831	2838	2831
query73	4176	311	311	311
query74	7180	6446	6421	6421
query75	5544	2327	2356	2327
query76	6463	1001	992	992
query77	809	226	229	226
query78	9088	8641	8654	8641
query79	2715	486	478	478
query80	646	318	306	306
query81	477	199	202	199
query82	210	83	81	81
query83	218	123	113	113
query84	275	68	67	67
query85	1107	337	326	326
query86	398	355	406	355
query87	3561	3328	3334	3328
query88	2969	2189	2184	2184
query89	423	363	359	359
query90	2268	193	185	185
query91	163	134	129	129
query92	48	44	41	41
query93	1539	425	442	425
query94	1548	162	157	157
query95	510	457	439	439
query96	614	320	314	314
query97	4262	4186	4149	4149
query98	206	193	181	181
query99	973	702	685	685
Total cold run time: 297666 ms
Total hot run time: 185839 ms

doris-robot · 2024-01-26T08:45:34Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66b6a556d542663e9086cf05c6c01aabb57fbc65, data reload: false
query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.22	0.04	0.06
query4	1.69	0.10	0.09
query5	0.54	0.52	0.54
query6	1.29	0.65	0.62
query7	0.01	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.50
query10	0.58	0.56	0.57
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.60	0.62
query14	0.79	0.80	0.80
query15	0.80	0.79	0.77
query16	0.38	0.39	0.39
query17	1.01	1.03	1.01
query18	0.26	0.25	0.25
query19	1.87	1.79	1.78
query20	0.01	0.01	0.01
query21	15.42	0.59	0.56
query22	2.48	2.64	1.63
query23	17.64	0.92	0.71
query24	2.44	0.71	1.44
query25	0.46	0.20	0.18
query26	0.49	0.13	0.14
query27	0.06	0.06	0.05
query28	11.69	0.77	0.75
query29	12.55	3.21	3.14
query30	0.56	0.50	0.48
query31	2.77	0.35	0.34
query32	3.37	0.50	0.48
query33	3.23	3.22	3.26
query34	16.01	4.28	4.29
query35	4.37	4.30	4.22
query36	1.10	1.08	1.07
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.01	0.02
Total cold run time: 106.01 s
Total hot run time: 30.4 s

doris-robot · 2024-01-26T08:48:39Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 66b6a556d542663e9086cf05c6c01aabb57fbc65 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.2 seconds inserted 10000000 Rows, about 704K ops/s

github-actions · 2024-01-26T10:14:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-01-26T10:14:59Z

PR approved by anyone and no changes requested.

Co-authored-by: Jerry Hu <mrhhsg@gmail.com>

### What problem does this PR solve? Related PR: #30133 Problem Summary: The original PR eliminate null aware with a wrong way to produce wrong result under some cases. This PR remove this optimization and will take by it in a correct way in next PR.

github-actionsBot added the meta-change label Jan 19, 2024

github-actionsBot reviewed Jan 19, 2024
View reviewed changes

mrhhsg force-pushed the nereids_markjoin branch from 46834c3 to a23b131Compare January 20, 2024 01:38

github-actionsBot reviewed Jan 20, 2024
View reviewed changes

mrhhsg force-pushed the nereids_markjoin branch from a23b131 to 5b4eb16Compare January 20, 2024 14:03

github-actionsBot reviewed Jan 21, 2024
View reviewed changes

starocean999 force-pushed the nereids_markjoin branch from d4988f8 to ff7c7faCompare January 21, 2024 11:37

github-actionsBot reviewed Jan 21, 2024
View reviewed changes

starocean999and others added 8 commits January 26, 2024 15:05

update out file
381d712

[be] support new mark join
3e3df2f

[fix] mark join without mark join conjuncts
b95789d

fix incorrect result of mark join
f9ab996

fix null aware join with other conjuncts
bfed82f

add test case
243bdc0
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>

fix code style
8a1fd36

modify based on comments
66b6a55

starocean999 force-pushed the nereids_markjoin branch from 0358f42 to 66b6a55Compare January 26, 2024 07:35

morrySnow approved these changes Jan 26, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 26, 2024

keanji-x approved these changes Jan 26, 2024
View reviewed changes

starocean999 merged commit 781477a into apache:masterJan 26, 2024

yiguolei pushed a commit that referenced this pull request Jan 27, 2024
[feature](nereids)support mark join (#30133)
713798d
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mrhhsg deleted the nereids_markjoin branch November 18, 2024 07:58

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[feature](nereids)support mark join (apache#30133)
7b92365
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>

This was referenced Jun 22, 2026
[dev](code-owner) add code owner for qe and statistics #64355
Merged
[fix](subquery) eliminate null aware by mistake #64639
Merged

	Statusdo_mark_join_conjuncts(Block*output_block, constsize_thash_table_bucket_size);
	Statusdo_mark_join_conjuncts(Block*output_block, size_thash_table_bucket_size);

	Status VExprContext::execute_conjuncts(const VExprContextSPtrs& conjuncts, Block* block,
	staticStatus VExprContext::execute_conjuncts(const VExprContextSPtrs& conjuncts, Block* block,

Uh oh!

Conversation

starocean999 commented Jan 19, 2024

Proposed changes

Further comments

Uh oh!

starocean999 commented Jan 19, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Jan 19, 2024

Uh oh!

starocean999 commented Jan 19, 2024

Uh oh!

doris-robot commented Jan 19, 2024

Uh oh!

starocean999 commented Jan 20, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 20, 2024

Choose a reason for hiding this comment

Uh oh!

mrhhsg commented Jan 20, 2024

Uh oh!

doris-robot commented Jan 20, 2024

Uh oh!

starocean999 commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

mrhhsg commented Jan 21, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 21, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

doris-robot commented Jan 21, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJan 21, 2024

Choose a reason for hiding this comment

Uh oh!

mrhhsg commented Jan 21, 2024

Uh oh!

github-actionsBot left a comment