Skip to content

[fix](ES Catalog)Do not push down limit to ES when predicates can not be processed by ES. - #40111

Merged
xiaokang merged 1 commit into
apache:masterfrom
qidaye:fix_pushdown_es
Sep 1, 2024
Merged

[fix](ES Catalog)Do not push down limit to ES when predicates can not be processed by ES.#40111
xiaokang merged 1 commit into
apache:masterfrom
qidaye:fix_pushdown_es

Conversation

@qidaye

@qidayeqidaye commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye

qidaye commented Aug 29, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
@github-actions

github-actionsBot commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38318 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c1bbe2972d22440a497e2b84f1dae239bcdcc575, data reload: false
------ Round 1 ----------------------------------
q1	19019	4677	4348	4348
q2	2022	192	168	168
q3	10482	1140	1185	1140
q4	10135	723	749	723
q5	7742	2933	2860	2860
q6	230	136	136	136
q7	971	609	600	600
q8	9325	2092	2087	2087
q9	7349	6578	6546	6546
q10	7002	2243	2217	2217
q11	476	238	249	238
q12	402	225	223	223
q13	17764	3055	3052	3052
q14	283	233	242	233
q15	538	485	474	474
q16	558	497	522	497
q17	978	673	734	673
q18	7545	6811	6800	6800
q19	1405	1082	1096	1082
q20	713	347	338	338
q21	4020	2964	2875	2875
q22	1116	1008	1035	1008
Total cold run time: 110075 ms
Total hot run time: 38318 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4384	4287	4224	4224
q2	377	270	276	270
q3	2909	2651	2671	2651
q4	1917	1674	1659	1659
q5	5421	5419	5429	5419
q6	221	129	130	129
q7	2146	1769	1765	1765
q8	3223	3373	3414	3373
q9	8538	8537	8462	8462
q10	3470	3195	3185	3185
q11	611	488	517	488
q12	812	627	620	620
q13	13754	3029	3007	3007
q14	300	265	278	265
q15	528	497	480	480
q16	635	570	563	563
q17	1783	1494	1496	1494
q18	7715	7572	7634	7572
q19	1667	1535	1544	1535
q20	2078	1838	1795	1795
q21	5457	5244	5186	5186
q22	1146	1059	1053	1053
Total cold run time: 69092 ms
Total hot run time: 55195 ms

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 188810 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c1bbe2972d22440a497e2b84f1dae239bcdcc575, data reload: false
query1	925	385	368	368
query2	6467	1990	1907	1907
query3	6652	214	229	214
query4	34346	23341	23142	23142
query5	4190	527	470	470
query6	263	186	179	179
query7	4593	294	295	294
query8	254	197	209	197
query9	8702	2482	2469	2469
query10	438	279	288	279
query11	17713	15066	15109	15066
query12	154	99	98	98
query13	1624	379	378	378
query14	9899	7721	7214	7214
query15	292	169	174	169
query16	8099	482	490	482
query17	1619	574	543	543
query18	2122	296	284	284
query19	330	142	148	142
query20	115	110	109	109
query21	207	104	101	101
query22	4537	4110	4239	4110
query23	34078	33826	33450	33450
query24	11202	2923	2919	2919
query25	649	402	413	402
query26	1164	163	162	162
query27	2650	286	285	285
query28	7421	2137	2114	2114
query29	834	432	421	421
query30	314	154	157	154
query31	1015	801	781	781
query32	102	65	67	65
query33	749	311	308	308
query34	908	502	491	491
query35	872	735	715	715
query36	1062	911	930	911
query37	167	103	105	103
query38	4041	3870	3926	3870
query39	1474	1393	1396	1393
query40	201	123	120	120
query41	50	49	47	47
query42	115	96	100	96
query43	493	480	477	477
query44	1261	755	754	754
query45	204	173	176	173
query46	1100	741	742	741
query47	1924	1792	1808	1792
query48	401	302	302	302
query49	1021	440	450	440
query50	819	414	419	414
query51	7230	7189	7018	7018
query52	99	88	91	88
query53	259	189	187	187
query54	885	457	467	457
query55	78	77	83	77
query56	283	279	262	262
query57	1211	1076	1069	1069
query58	249	238	261	238
query59	2937	2805	2695	2695
query60	308	282	280	280
query61	123	120	128	120
query62	800	662	662	662
query63	222	189	183	183
query64	5083	782	749	749
query65	3246	3161	3187	3161
query66	1300	342	353	342
query67	15520	15317	15279	15279
query68	4627	590	579	579
query69	410	290	279	279
query70	1130	1110	1021	1021
query71	333	284	277	277
query72	6792	4090	3955	3955
query73	763	337	344	337
query74	9360	8930	8886	8886
query75	3393	2651	2687	2651
query76	2864	1200	1002	1002
query77	490	330	319	319
query78	9817	9187	9388	9187
query79	2528	549	557	549
query80	1134	511	504	504
query81	593	234	232	232
query82	674	144	141	141
query83	216	153	152	152
query84	237	78	77	77
query85	1717	295	323	295
query86	373	298	287	287
query87	4365	4313	4255	4255
query88	4135	2362	2335	2335
query89	404	291	287	287
query90	1698	191	191	191
query91	129	121	100	100
query92	67	52	55	52
query93	2221	569	551	551
query94	640	299	297	297
query95	346	299	252	252
query96	606	271	267	267
query97	3192	3037	3088	3037
query98	224	207	212	207
query99	1554	1289	1267	1267
Total cold run time: 296697 ms
Total hot run time: 188810 ms

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c1bbe2972d22440a497e2b84f1dae239bcdcc575, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.08
query5	0.48	0.50	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.54	0.55
query11	0.15	0.12	0.11
query12	0.15	0.13	0.13
query13	0.61	0.58	0.58
query14	2.10	2.03	2.04
query15	0.90	0.80	0.82
query16	0.36	0.37	0.37
query17	1.05	1.03	0.99
query18	0.21	0.20	0.20
query19	1.94	1.85	1.90
query20	0.01	0.00	0.01
query21	15.39	0.66	0.66
query22	4.70	6.24	2.47
query23	18.31	1.38	1.22
query24	2.09	0.22	0.22
query25	0.15	0.08	0.07
query26	0.27	0.18	0.17
query27	0.08	0.07	0.07
query28	13.27	1.02	0.99
query29	12.62	3.31	3.29
query30	0.24	0.06	0.05
query31	2.87	0.39	0.40
query32	3.25	0.48	0.46
query33	2.96	3.02	3.01
query34	16.92	4.40	4.42
query35	4.47	4.46	4.45
query36	0.65	0.47	0.46
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.05
query40	0.16	0.12	0.12
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.26 s
Total hot run time: 32.54 s

morningman
morningman approved these changes Aug 29, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang
xiaokang merged commit f87e51c into apache:masterSep 1, 2024
@xiaokangxiaokang removed the doing label Sep 1, 2024
@qidaye
qidaye deleted the fix_pushdown_es branch September 2, 2024 02:29
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 2, 2024
@qidaye
[fix](ES Catalog)Do not push down limit to ES when predicates can not…
715efb4
… be processed by ES. (apache#40111)
@qidayeqidaye mentioned this pull request Sep 2, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 2, 2024
@qidaye
[fix](ES Catalog)Do not push down limit to ES when predicates can not…
ee2b51b
… be processed by ES. (apache#40111)
@qidayeqidaye mentioned this pull request Sep 2, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 3, 2024
@qidaye
[fix](ES Catalog)Do not push down limit to ES when predicates can not…
41271ec
… be processed by ES. (#40111) (#40265)
bp #40111
xiaokang pushed a commit that referenced this pull request Sep 3, 2024
@qidaye
[fix](ES Catalog)Do not push down limit to ES when predicates can not…
abd9ae2
… be processed by ES. (#40111) (#40266)
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@qidaye@dataroaring
[fix](ES Catalog)Do not push down limit to ES when predicates can not…
6dacb0e
… be processed by ES. (#40111)
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@qidaye@HappenLee
[fix](ES Catalog)Do not push down limit to ES when predicates can not…
0671ef4
… be processed by ES. (apache#40111)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@Gabriel39Gabriel39Gabriel39 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.15-mergeddev/2.1.6-mergeddev/3.0.2-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@qidaye@doris-robot@morningman@Gabriel39@xiaokang@yiguolei@dataroaring