[feature](ES Catalog)Support control scroll level by config by qidaye · Pull Request #37180 · apache/doris

qidaye · 2024-07-02T14:23:16Z

Proposed changes

Add session variable enable_es_parallel_scroll, default true. When it's false, Doris will send a scroll query to ES by one backend without shard preference and sort info. Which equals to call the search api directly of ES and it's very useful when you need a global sorting based on scores.
Print ES response body when http request is not OK.

doris-robot · 2024-07-02T14:23:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

qidaye · 2024-07-02T14:25:33Z

run buildall

github-actions · 2024-07-02T14:29:02Z

clang-tidy review says "All clean, LGTM! 👍"

qidaye · 2024-07-02T14:35:33Z

run buildall

github-actions · 2024-07-02T14:40:49Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-02T15:31:03Z

TPC-H: Total hot run time: 39709 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5665dcbf6cd0f1c9af87f149d33e98f267d97adf, data reload: false
------ Round 1 ----------------------------------
q1	17604	4666	4303	4303
q2	2016	195	191	191
q3	10520	1188	1152	1152
q4	10203	770	736	736
q5	7489	2673	2609	2609
q6	216	135	135	135
q7	946	591	611	591
q8	9217	2087	2070	2070
q9	8984	6481	6540	6481
q10	8881	3694	3751	3694
q11	461	232	236	232
q12	472	232	224	224
q13	17767	2955	2966	2955
q14	276	219	233	219
q15	524	489	505	489
q16	530	374	360	360
q17	972	668	628	628
q18	8159	7373	7417	7373
q19	7777	1549	1496	1496
q20	660	313	317	313
q21	4897	3126	3320	3126
q22	390	337	332	332
Total cold run time: 118961 ms
Total hot run time: 39709 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4400	4255	4249	4249
q2	379	278	276	276
q3	2991	2837	2882	2837
q4	2011	1678	1781	1678
q5	5502	5481	5458	5458
q6	236	136	131	131
q7	2242	1885	1860	1860
q8	3252	3455	3439	3439
q9	8719	8691	8737	8691
q10	4175	3727	3782	3727
q11	615	504	508	504
q12	813	663	643	643
q13	16947	3181	3192	3181
q14	313	273	263	263
q15	521	480	511	480
q16	492	437	408	408
q17	1806	1545	1501	1501
q18	8215	7927	8004	7927
q19	1886	1695	1625	1625
q20	2169	1868	1882	1868
q21	5115	4724	4938	4724
q22	648	537	554	537
Total cold run time: 73447 ms
Total hot run time: 56007 ms

doris-robot · 2024-07-02T15:41:57Z

TPC-DS: Total hot run time: 172748 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5665dcbf6cd0f1c9af87f149d33e98f267d97adf, data reload: false
query1	916	380	372	372
query2	6447	2473	2355	2355
query3	6639	204	214	204
query4	19002	17524	17649	17524
query5	3624	478	453	453
query6	268	165	167	165
query7	4597	298	287	287
query8	321	293	288	288
query9	8449	2337	2326	2326
query10	567	301	273	273
query11	10445	9884	9849	9849
query12	115	80	78	78
query13	1653	366	363	363
query14	10064	7734	6167	6167
query15	223	174	188	174
query16	7984	267	279	267
query17	1895	532	513	513
query18	2047	273	269	269
query19	185	144	146	144
query20	85	79	80	79
query21	206	149	130	130
query22	4281	4036	4001	4001
query23	33998	33658	33794	33658
query24	10852	2890	2848	2848
query25	579	367	368	367
query26	693	151	151	151
query27	2246	334	322	322
query28	6035	2135	2138	2135
query29	902	616	654	616
query30	258	162	153	153
query31	977	764	775	764
query32	100	55	57	55
query33	650	276	289	276
query34	898	470	486	470
query35	749	653	624	624
query36	1104	1026	958	958
query37	139	79	77	77
query38	2964	2890	2797	2797
query39	884	849	835	835
query40	214	130	125	125
query41	59	50	53	50
query42	119	100	104	100
query43	571	558	572	558
query44	1114	755	743	743
query45	199	158	165	158
query46	1075	717	684	684
query47	1818	1766	1740	1740
query48	369	311	321	311
query49	877	434	404	404
query50	753	376	373	373
query51	6921	6741	6797	6741
query52	105	97	91	91
query53	352	284	286	284
query54	886	442	439	439
query55	74	74	70	70
query56	281	261	264	261
query57	1132	1060	1014	1014
query58	256	237	238	237
query59	3324	3038	3145	3038
query60	295	273	269	269
query61	92	91	88	88
query62	589	456	448	448
query63	311	289	285	285
query64	8544	2241	1737	1737
query65	3139	3093	3068	3068
query66	752	327	326	326
query67	15463	15025	14736	14736
query68	4613	533	549	533
query69	581	395	339	339
query70	1201	1127	1168	1127
query71	375	278	269	269
query72	7044	5177	5825	5177
query73	759	327	328	327
query74	5861	5529	5577	5529
query75	3381	2718	2691	2691
query76	2170	941	941	941
query77	437	300	294	294
query78	10508	9919	10062	9919
query79	2274	518	513	513
query80	2237	522	502	502
query81	572	225	228	225
query82	1029	116	110	110
query83	308	175	176	175
query84	266	102	97	97
query85	1213	332	320	320
query86	463	330	322	322
query87	3228	3069	3036	3036
query88	3213	2464	2455	2455
query89	476	402	389	389
query90	1715	189	190	189
query91	138	110	110	110
query92	67	50	52	50
query93	1920	518	515	515
query94	1153	202	196	196
query95	423	324	325	324
query96	583	270	267	267
query97	3160	3006	3035	3006
query98	219	197	202	197
query99	1184	854	839	839
Total cold run time: 266496 ms
Total hot run time: 172748 ms

doris-robot · 2024-07-02T15:47:28Z

ClickBench: Total hot run time: 30.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5665dcbf6cd0f1c9af87f149d33e98f267d97adf, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.07	0.08
query5	0.49	0.48	0.47
query6	1.15	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.48	0.48
query10	0.54	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.75	0.79	0.78
query15	0.85	0.82	0.81
query16	0.38	0.37	0.34
query17	1.02	0.98	0.96
query18	0.22	0.21	0.23
query19	1.90	1.65	1.73
query20	0.02	0.01	0.01
query21	15.42	0.74	0.66
query22	4.38	7.02	1.87
query23	18.22	1.36	1.19
query24	2.11	0.24	0.21
query25	0.15	0.08	0.08
query26	0.27	0.17	0.17
query27	0.09	0.08	0.07
query28	13.21	1.01	0.99
query29	12.63	3.33	3.27
query30	0.26	0.06	0.05
query31	2.85	0.39	0.40
query32	3.27	0.47	0.47
query33	2.84	2.94	2.91
query34	17.19	4.42	4.43
query35	4.46	4.49	4.55
query36	0.65	0.48	0.48
query37	0.19	0.16	0.16
query38	0.17	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.15
query41	0.08	0.04	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.58 s
Total hot run time: 30.35 s

qidaye · 2024-07-03T03:35:11Z

run external

qidaye · 2024-07-03T03:35:15Z

run p0

qidaye · 2024-07-03T06:10:04Z

run buildall

github-actions · 2024-07-03T06:15:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T06:34:46Z

TPC-H: Total hot run time: 39651 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 957e05ef831e717accc30bf160d3504b87ccfc72, data reload: false
------ Round 1 ----------------------------------
q1	17600	5196	4214	4214
q2	2006	202	193	193
q3	10433	1290	1092	1092
q4	10203	758	893	758
q5	7524	2706	2630	2630
q6	229	141	142	141
q7	965	609	608	608
q8	9552	2084	2087	2084
q9	8723	6470	6483	6470
q10	9025	3686	3743	3686
q11	464	240	240	240
q12	440	245	228	228
q13	17777	2959	3011	2959
q14	262	237	219	219
q15	536	494	493	493
q16	516	371	371	371
q17	971	725	673	673
q18	8020	7430	7409	7409
q19	5142	1408	1472	1408
q20	671	327	325	325
q21	4891	3112	3938	3112
q22	397	338	361	338
Total cold run time: 116347 ms
Total hot run time: 39651 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4406	4208	4266	4208
q2	369	274	270	270
q3	2968	2741	2703	2703
q4	1885	1642	1674	1642
q5	5248	5290	5277	5277
q6	223	130	133	130
q7	2101	1764	1750	1750
q8	3187	3344	3313	3313
q9	8343	8310	8353	8310
q10	3862	3680	3649	3649
q11	583	489	491	489
q12	801	588	605	588
q13	16664	3035	2969	2969
q14	296	255	265	255
q15	501	483	478	478
q16	476	407	432	407
q17	1762	1465	1464	1464
q18	7698	7630	7402	7402
q19	3466	1631	1672	1631
q20	1998	1827	1761	1761
q21	4851	4648	4822	4648
q22	637	547	544	544
Total cold run time: 72325 ms
Total hot run time: 53888 ms

doris-robot · 2024-07-03T06:45:47Z

TPC-DS: Total hot run time: 172839 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 957e05ef831e717accc30bf160d3504b87ccfc72, data reload: false
query1	919	382	365	365
query2	6473	2374	2385	2374
query3	6652	204	213	204
query4	19841	17707	17193	17193
query5	4166	488	484	484
query6	271	185	177	177
query7	4566	294	297	294
query8	327	316	300	300
query9	8246	2391	2373	2373
query10	611	312	287	287
query11	10567	10155	10140	10140
query12	128	85	80	80
query13	1589	365	372	365
query14	10224	7628	7123	7123
query15	224	183	186	183
query16	7779	266	263	263
query17	1906	533	519	519
query18	1931	274	290	274
query19	201	151	158	151
query20	87	80	83	80
query21	221	127	133	127
query22	4213	3985	3953	3953
query23	34059	33023	33258	33023
query24	11129	2767	2807	2767
query25	629	353	363	353
query26	1563	151	151	151
query27	2915	311	314	311
query28	7499	2060	2075	2060
query29	1016	649	609	609
query30	289	151	154	151
query31	954	743	748	743
query32	92	53	53	53
query33	757	289	289	289
query34	938	467	466	466
query35	750	594	634	594
query36	1149	909	933	909
query37	152	74	75	74
query38	2877	2773	2777	2773
query39	849	795	794	794
query40	261	129	125	125
query41	55	53	51	51
query42	120	97	100	97
query43	587	571	554	554
query44	1221	730	736	730
query45	189	162	159	159
query46	1078	732	739	732
query47	1849	1771	1759	1759
query48	378	310	306	306
query49	1182	404	404	404
query50	787	409	386	386
query51	6880	6902	6766	6766
query52	103	101	92	92
query53	372	295	292	292
query54	926	450	449	449
query55	77	76	71	71
query56	279	268	264	264
query57	1177	1068	1025	1025
query58	255	239	249	239
query59	3367	3154	3119	3119
query60	313	292	285	285
query61	93	93	116	93
query62	656	446	491	446
query63	322	292	292	292
query64	9966	2267	1790	1790
query65	3174	3087	3105	3087
query66	1375	344	328	328
query67	15433	15088	15027	15027
query68	4571	545	540	540
query69	465	312	301	301
query70	1180	1087	1148	1087
query71	381	283	275	275
query72	7187	5202	5519	5202
query73	735	317	319	317
query74	5906	5524	5501	5501
query75	3368	2638	2669	2638
query76	2474	888	862	862
query77	457	303	300	300
query78	10222	9894	9686	9686
query79	2336	513	509	509
query80	985	473	462	462
query81	588	217	221	217
query82	750	109	103	103
query83	233	168	167	167
query84	235	91	83	83
query85	2001	268	264	264
query86	512	332	322	322
query87	3255	3115	3081	3081
query88	4199	2489	2457	2457
query89	474	373	374	373
query90	1846	185	184	184
query91	129	99	105	99
query92	69	47	48	47
query93	2450	513	515	513
query94	1252	186	187	186
query95	400	317	316	316
query96	610	270	266	266
query97	3240	3081	3076	3076
query98	208	204	196	196
query99	1117	829	812	812
Total cold run time: 275069 ms
Total hot run time: 172839 ms

doris-robot · 2024-07-03T06:51:19Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 957e05ef831e717accc30bf160d3504b87ccfc72, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.09
query5	0.50	0.49	0.50
query6	1.14	0.73	0.74
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.53	0.49	0.49
query10	0.55	0.54	0.55
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.77	0.80	0.76
query15	0.84	0.82	0.81
query16	0.35	0.38	0.37
query17	0.99	0.95	0.95
query18	0.19	0.28	0.24
query19	1.84	1.67	1.71
query20	0.02	0.01	0.01
query21	15.45	0.72	0.66
query22	4.10	8.22	1.82
query23	18.30	1.43	1.27
query24	2.13	0.23	0.22
query25	0.16	0.08	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.23	1.02	1.00
query29	12.58	3.27	3.30
query30	0.25	0.06	0.06
query31	2.88	0.39	0.39
query32	3.29	0.47	0.46
query33	2.92	2.91	2.89
query34	17.13	4.44	4.42
query35	4.49	4.44	4.50
query36	0.64	0.46	0.50
query37	0.18	0.16	0.15
query38	0.16	0.14	0.16
query39	0.04	0.04	0.03
query40	0.17	0.14	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.34 s
Total hot run time: 30.4 s

1. Add config `enable_es_shard_scroll`, default true.

qidaye · 2024-07-03T10:17:13Z

run buildall

qidaye · 2024-07-03T10:19:01Z

run buildall

github-actions · 2024-07-03T10:22:45Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-03T10:24:43Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T10:36:34Z

TPC-H: Total hot run time: 40724 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f51e1e3ee8c677d0e2ad4423a60c6b6adcbbbad0, data reload: false
------ Round 1 ----------------------------------
q1	17839	4540	4413	4413
q2	2652	199	197	197
q3	11127	1217	1236	1217
q4	10590	891	817	817
q5	7925	2698	2627	2627
q6	222	137	141	137
q7	987	611	612	611
q8	9257	2081	2078	2078
q9	8782	6486	6464	6464
q10	8950	3708	3714	3708
q11	447	243	237	237
q12	419	241	233	233
q13	17762	2977	2979	2977
q14	279	228	217	217
q15	521	484	494	484
q16	508	381	374	374
q17	969	683	710	683
q18	8004	7578	7427	7427
q19	7848	1589	1328	1328
q20	648	323	327	323
q21	4950	3840	3834	3834
q22	386	338	338	338
Total cold run time: 121072 ms
Total hot run time: 40724 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4427	4280	4315	4280
q2	372	275	284	275
q3	2986	2761	2719	2719
q4	1880	1598	1594	1594
q5	5274	5278	5250	5250
q6	228	131	131	131
q7	2159	1790	1760	1760
q8	3177	3335	3290	3290
q9	8360	8326	8388	8326
q10	3844	3709	3633	3633
q11	577	495	511	495
q12	795	642	598	598
q13	16417	2996	3016	2996
q14	297	274	268	268
q15	512	477	519	477
q16	466	411	435	411
q17	1795	1478	1469	1469
q18	7613	7534	7387	7387
q19	5669	1512	1499	1499
q20	1977	1739	1728	1728
q21	4899	4682	4700	4682
q22	613	548	540	540
Total cold run time: 74337 ms
Total hot run time: 53808 ms

doris-robot · 2024-07-03T10:47:41Z

TPC-DS: Total hot run time: 171883 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f51e1e3ee8c677d0e2ad4423a60c6b6adcbbbad0, data reload: false
query1	922	407	365	365
query2	6464	2300	2365	2300
query3	6654	210	213	210
query4	22447	17297	17359	17297
query5	4170	486	495	486
query6	274	185	179	179
query7	4598	299	295	295
query8	317	288	313	288
query9	8774	2387	2368	2368
query10	612	319	291	291
query11	10458	10066	10061	10061
query12	133	90	86	86
query13	1660	360	369	360
query14	10266	6328	7498	6328
query15	245	184	190	184
query16	7767	277	273	273
query17	1893	538	537	537
query18	1947	283	267	267
query19	197	147	147	147
query20	90	82	80	80
query21	209	132	127	127
query22	4219	3995	3893	3893
query23	33828	33100	33096	33096
query24	11953	2823	2750	2750
query25	689	366	367	366
query26	1870	154	152	152
query27	2968	314	321	314
query28	7517	2074	2046	2046
query29	1145	607	609	607
query30	273	151	155	151
query31	966	736	734	734
query32	95	52	60	52
query33	759	297	291	291
query34	923	473	469	469
query35	748	618	596	596
query36	1115	981	904	904
query37	195	78	75	75
query38	2874	2715	2796	2715
query39	868	791	813	791
query40	271	130	122	122
query41	55	52	50	50
query42	115	99	98	98
query43	588	571	551	551
query44	1157	742	737	737
query45	194	165	162	162
query46	1072	711	734	711
query47	1828	1770	1759	1759
query48	372	297	306	297
query49	1215	415	413	413
query50	780	404	401	401
query51	6898	6742	6705	6705
query52	103	95	93	93
query53	360	294	304	294
query54	1009	467	448	448
query55	75	74	73	73
query56	276	265	270	265
query57	1194	1047	1042	1042
query58	284	233	238	233
query59	3295	3082	2950	2950
query60	304	279	275	275
query61	91	89	89	89
query62	654	456	448	448
query63	323	290	299	290
query64	9857	2283	1815	1815
query65	3181	3112	3122	3112
query66	1362	337	337	337
query67	15292	15035	14848	14848
query68	6273	551	556	551
query69	611	413	343	343
query70	1137	1117	1165	1117
query71	466	279	281	279
query72	7225	5454	5300	5300
query73	777	331	342	331
query74	5836	5562	5500	5500
query75	3899	2672	2654	2654
query76	3713	1032	917	917
query77	661	311	301	301
query78	10256	9813	9760	9760
query79	2044	517	519	517
query80	798	488	468	468
query81	559	226	220	220
query82	1457	108	106	106
query83	190	173	170	170
query84	249	92	86	86
query85	1408	276	265	265
query86	462	312	288	288
query87	3313	3052	3109	3052
query88	3985	2460	2441	2441
query89	468	399	381	381
query90	1863	191	194	191
query91	126	98	103	98
query92	61	48	50	48
query93	2183	522	532	522
query94	1169	188	189	188
query95	412	334	316	316
query96	597	269	270	269
query97	3161	3033	3063	3033
query98	220	202	198	198
query99	1151	824	828	824
Total cold run time: 282037 ms
Total hot run time: 171883 ms

doris-robot · 2024-07-03T10:53:12Z

ClickBench: Total hot run time: 30.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f51e1e3ee8c677d0e2ad4423a60c6b6adcbbbad0, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.10	0.10
query5	0.49	0.48	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.52	0.56	0.55
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.60	0.58
query14	0.78	0.78	0.80
query15	0.84	0.82	0.81
query16	0.37	0.37	0.35
query17	0.97	1.01	1.05
query18	0.20	0.25	0.27
query19	1.81	1.70	1.71
query20	0.02	0.01	0.01
query21	15.44	0.77	0.66
query22	4.27	7.84	1.61
query23	18.29	1.34	1.33
query24	2.20	0.22	0.24
query25	0.15	0.10	0.09
query26	0.26	0.18	0.18
query27	0.08	0.08	0.08
query28	13.25	1.01	1.01
query29	12.69	3.39	3.37
query30	0.25	0.06	0.05
query31	2.88	0.40	0.39
query32	3.24	0.48	0.47
query33	2.88	2.95	2.87
query34	17.00	4.39	4.37
query35	4.52	4.50	4.55
query36	0.65	0.46	0.50
query37	0.18	0.15	0.16
query38	0.16	0.15	0.14
query39	0.05	0.03	0.03
query40	0.20	0.15	0.15
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.47 s
Total hot run time: 30.48 s

1. Add config `enable_es_shard_scroll`, default true. 2. Print ES response body when http request is not OK.

xiaokang

LGTM

github-actions · 2024-07-09T13:02:52Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-09T13:02:54Z

PR approved by anyone and no changes requested.

morningman

LGTM

…7180) ## Proposed changes 1. Add session variable `enable_es_parallel_scroll`, default true. When it's false, Doris will send a scroll query to ES by one backend without shard preference and sort info. Which equals to call the search api directly of ES and it's very useful when you need a global sorting based on scores. 2. Print ES response body when http request is not OK.

…37290) ## Proposed changes backport #37180

…37271) backport #37180

…7180) ## Proposed changes 1. Add session variable `enable_es_parallel_scroll`, default true. When it's false, Doris will send a scroll query to ES by one backend without shard preference and sort info. Which equals to call the search api directly of ES and it's very useful when you need a global sorting based on scores. 2. Print ES response body when http request is not OK.

## Proposed changes 1. Add session variable `enable_es_parallel_scroll`, default true. When it's false, Doris will send a scroll query to ES by one backend without shard preference and sort info. Which equals to call the search api directly of ES and it's very useful when you need a global sorting based on scores. 2. Print ES response body when http request is not OK.

1. Add json type mapping apache/doris#37101 2. Add session instruction `enable_es_parallel_scroll` and `batch_size` apache/doris#37180

(apache#37271) backport apache#37180

1. Add json type mapping apache/doris#37101 2. Add session instruction `enable_es_parallel_scroll` and `batch_size` apache/doris#37180

…7180) ## Proposed changes 1. Add session variable `enable_es_parallel_scroll`, default true. When it's false, Doris will send a scroll query to ES by one backend without shard preference and sort info. Which equals to call the search api directly of ES and it's very useful when you need a global sorting based on scores. 2. Print ES response body when http request is not OK.

qidaye force-pushed the es_scroll_thread_config branch from d953593 to 5665dcbCompare July 2, 2024 14:35

qidaye force-pushed the es_scroll_thread_config branch from 5665dcb to 957e05eCompare July 3, 2024 06:09

[feature](ES Catalog)Support control scroll level by config
2bd10ae
1. Add config `enable_es_shard_scroll`, default true.

qidaye force-pushed the es_scroll_thread_config branch from 957e05e to 6de7a2dCompare July 3, 2024 10:17

print ES error response body to log
f51e1e3

qidaye force-pushed the es_scroll_thread_config branch from 6de7a2d to f51e1e3Compare July 3, 2024 10:18

qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 4, 2024
[feature](ES Catalog)Support control scroll level by configapache#37180
a878f60
1. Add config `enable_es_shard_scroll`, default true. 2. Print ES response body when http request is not OK.

qidaye mentioned this pull request Jul 4, 2024
[feature](ES Catalog)Support control scroll level by config #37180 #37271
Merged

xiaokang added dev/2.1.x and removed dev/2.0.x dev/2.1.x labels Jul 9, 2024

xiaokang approved these changes Jul 9, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2024

github-actionsBot added the reviewed label Jul 9, 2024

morningman approved these changes Jul 12, 2024
View reviewed changes

morningman merged commit a28f451 into apache:masterJul 12, 2024

morningman added the dev/2.1.x label Jul 12, 2024

qidaye deleted the es_scroll_thread_config branch July 15, 2024 03:05

morningman pushed a commit that referenced this pull request Jul 15, 2024
[feature](ES Catalog)Support control scroll level by config#37180(#…
e5339a4
…37290) ## Proposed changes backport #37180

morningman added dev/2.1.5-merged and removed dev/2.1.x labels Jul 15, 2024

morningman pushed a commit that referenced this pull request Jul 15, 2024
[feature](ES Catalog)Support control scroll level by config#37180(#…
4acc692
…37271) backport #37180

morningman added dev/2.0.13-merged dev/2.0.14-merged and removed dev/2.0.13-merged labels Jul 15, 2024

dataroaring added dev/3.0.1-merged and removed dev/3.0.x labels Jul 25, 2024

qidaye mentioned this pull request Aug 2, 2024
[doc](ES Catalog) Update ES Catalog docs apache/doris-website#943
Merged

morningman pushed a commit to apache/doris-website that referenced this pull request Aug 5, 2024
[doc](ES Catalog) Update ES Catalog docs (#943)
e6eb4f8
1. Add json type mapping apache/doris#37101 2. Add session instruction `enable_es_parallel_scroll` and `batch_size` apache/doris#37180

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[feature](ES Catalog)Support control scroll level by configapache#37180
fcaf20c
(apache#37271) backport apache#37180

gavinchou mentioned this pull request Aug 19, 2024
Release Note 3.0.1 #39570
Closed

Gabriel39 pushed a commit to Gabriel39/doris-website that referenced this pull request Sep 23, 2024
[doc](ES Catalog) Update ES Catalog docs (apache#943)
06fe8f2
1. Add json type mapping apache/doris#37101 2. Add session instruction `enable_es_parallel_scroll` and `batch_size` apache/doris#37180

Uh oh!

Conversation

qidaye commented Jul 2, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

qidaye commented Jul 2, 2024

Uh oh!

github-actionsBot commented Jul 2, 2024

Uh oh!

qidaye commented Jul 2, 2024

Uh oh!

github-actionsBot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

qidaye commented Jul 3, 2024

Uh oh!

qidaye commented Jul 3, 2024

Uh oh!

qidaye commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

qidaye commented Jul 3, 2024

Uh oh!

qidaye commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 9, 2024

Uh oh!

github-actionsBot commented Jul 9, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

qidaye commented Jul 2, 2024•
edited
Loading