Skip to content

[fix](maxcompute)Fix MaxCompute IN predicate pushdown polarity. - #65083

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
hubgeter:fix_mc_in_master
Jul 6, 2026
Merged

[fix](maxcompute)Fix MaxCompute IN predicate pushdown polarity.#65083
Gabriel39 merged 1 commit into
apache:masterfrom
hubgeter:fix_mc_in_master

Conversation

@hubgeter

@hubgeterhubgeter commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:
fix maxcompute error in predicate pushdown.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

hubgeter commented Jul 1, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hubgeterhubgeter mentioned this pull request Jul 1, 2026
Merged
16 tasks
@hello-stephen

hello-stephen commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Jul 1, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29429 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19f6e37c2d4878356acb4191e5870c9a4fa6ef6e, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17795	4206	4061	4061
q2	2034	345	194	194
q3	10247	1408	850	850
q4	4724	470	341	341
q5	7582	865	574	574
q6	216	173	139	139
q7	797	863	620	620
q8	10493	1502	1503	1502
q9	5798	4469	4445	4445
q10	6817	1809	1527	1527
q11	514	349	316	316
q12	705	574	433	433
q13	18083	3369	2757	2757
q14	270	263	238	238
q15	q16	782	784	716	716
q17	979	1056	942	942
q18	6733	5777	5479	5479
q19	1178	1336	1069	1069
q20	755	647	560	560
q21	5670	2684	2370	2370
q22	433	362	296	296
Total cold run time: 102605 ms
Total hot run time: 29429 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4405	4251	4248	4248
q2	292	345	212	212
q3	4567	4941	4435	4435
q4	2099	2183	1379	1379
q5	4433	4323	4274	4274
q6	230	178	130	130
q7	1742	2201	1668	1668
q8	2465	2186	2135	2135
q9	7949	7856	7783	7783
q10	4778	4743	4284	4284
q11	571	428	375	375
q12	748	878	609	609
q13	3280	3533	3025	3025
q14	319	300	279	279
q15	q16	764	734	647	647
q17	1343	1337	1329	1329
q18	8048	7355	6800	6800
q19	1134	1100	1099	1099
q20	2236	2227	1936	1936
q21	5266	4580	4447	4447
q22	537	468	427	427
Total cold run time: 57206 ms
Total hot run time: 51521 ms

@hello-stephen

hello-stephen commented Jul 1, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 174228 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19f6e37c2d4878356acb4191e5870c9a4fa6ef6e, data reload: false
query5	4337	639	500	500
query6	463	239	203	203
query7	4832	614	329	329
query8	355	191	169	169
query9	8782	4046	4055	4046
query10	488	351	308	308
query11	5915	2372	2157	2157
query12	165	107	101	101
query13	1281	615	432	432
query14	6272	5314	4978	4978
query14_1	4341	4355	4345	4345
query15	221	204	181	181
query16	1051	468	468	468
query17	1128	744	582	582
query18	2499	487	378	378
query19	205	193	170	170
query20	111	119	110	110
query21	236	155	135	135
query22	13660	13587	13428	13428
query23	17432	16473	16248	16248
query23_1	16279	16148	16291	16148
query24	7477	1789	1276	1276
query24_1	1307	1310	1291	1291
query25	566	486	388	388
query26	1329	342	216	216
query27	2604	597	383	383
query28	4475	2057	2039	2039
query29	1080	632	501	501
query30	339	263	228	228
query31	1116	1095	981	981
query32	121	65	63	63
query33	542	333	267	267
query34	1201	1142	680	680
query35	781	801	675	675
query36	1440	1412	1255	1255
query37	150	106	96	96
query38	1887	1718	1654	1654
query39	928	940	885	885
query39_1	882	883	906	883
query40	250	164	143	143
query41	66	63	63	63
query42	98	90	93	90
query43	320	320	281	281
query44	1466	776	770	770
query45	200	188	179	179
query46	1106	1163	718	718
query47	2352	2354	2247	2247
query48	410	418	297	297
query49	677	418	317	317
query50	1053	424	338	338
query51	4381	4325	4358	4325
query52	85	87	76	76
query53	276	267	210	210
query54	299	229	210	210
query55	74	70	66	66
query56	310	273	286	273
query57	1423	1414	1307	1307
query58	301	259	277	259
query59	1644	1639	1421	1421
query60	297	269	236	236
query61	151	152	147	147
query62	717	650	575	575
query63	238	206	205	205
query64	2523	775	600	600
query65	4885	4797	4796	4796
query66	1819	508	384	384
query67	30041	29753	29640	29640
query68	3212	1574	1042	1042
query69	458	303	272	272
query70	1083	944	959	944
query71	355	335	313	313
query72	2969	2700	2323	2323
query73	834	771	428	428
query74	5121	4961	4751	4751
query75	2625	2584	2225	2225
query76	2379	1190	794	794
query77	347	389	284	284
query78	12530	12552	11926	11926
query79	1400	1152	770	770
query80	1317	555	459	459
query81	557	324	272	272
query82	645	163	118	118
query83	360	313	297	297
query84	288	166	133	133
query85	966	620	531	531
query86	426	293	277	277
query87	1833	1837	1774	1774
query88	3720	2812	2806	2806
query89	459	409	350	350
query90	2042	198	197	197
query91	201	191	160	160
query92	66	59	60	59
query93	1767	1596	1006	1006
query94	736	349	323	323
query95	789	592	488	488
query96	1026	838	357	357
query97	2702	2673	2549	2549
query98	224	211	201	201
query99	1201	1152	1021	1021
Total cold run time: 260869 ms
Total hot run time: 174228 ms

@hello-stephen

hello-stephen commented Jul 1, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19f6e37c2d4878356acb4191e5870c9a4fa6ef6e, data reload: false
query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.14	0.13
query4	1.60	0.17	0.14
query5	0.24	0.22	0.22
query6	1.26	1.07	1.07
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.38	0.31	0.31
query10	0.56	0.55	0.57
query11	0.20	0.14	0.14
query12	0.18	0.15	0.15
query13	0.47	0.47	0.48
query14	1.02	1.01	0.99
query15	0.61	0.59	0.60
query16	0.31	0.31	0.31
query17	1.09	1.10	1.10
query18	0.22	0.21	0.20
query19	2.09	1.96	2.00
query20	0.02	0.01	0.02
query21	15.46	0.22	0.14
query22	4.89	0.06	0.05
query23	16.15	0.31	0.12
query24	2.93	0.40	0.33
query25	0.12	0.05	0.04
query26	0.75	0.22	0.16
query27	0.04	0.03	0.04
query28	3.47	0.90	0.55
query29	12.54	4.39	3.46
query30	0.28	0.15	0.15
query31	2.79	0.59	0.31
query32	3.24	0.59	0.49
query33	3.25	3.22	3.18
query34	15.61	4.21	3.56
query35	3.53	3.46	3.53
query36	0.55	0.43	0.42
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.85 s
Total hot run time: 25.26 s

@Gabriel39

Gabriel39 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

/review

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 6, 2026
@github-actions

github-actionsBot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

github-actions[bot]
github-actionsBot previously requested changes Jul 6, 2026
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary:

I found one blocking issue: the changed MaxCompute IN/NOT IN polarity branch is not covered by local FE tests, and the current PR head still has check_coverage_fe (Coverage) failing. The end-to-end MaxCompute regression is useful, but it is remote-gated and does not satisfy the FE coverage job.

Critical checkpoints:

  • Goal: fixes MaxCompute IN predicate pushdown polarity; code maps non-negated IN to ODPS IN and NOT IN to ODPS NOT_IN.
  • Scope: small and focused, limited to the converter plus a regression suite/output.
  • Parallel paths: the newer connector MaxCompute predicate converter already uses matching polarity; no additional parallel-path issue found.
  • Concurrency/lifecycle/persistence/config/protocol: not materially changed.
  • Tests: remote regression added and expected output is ordered; local FE coverage for the changed Java branch is missing, matching the failing coverage check.
  • Validation: static review plus GitHub PR metadata/checks; local FE build/tests were not run because this runner lacks thirdparty/installed/bin/protoc.

Subagent conclusions: optimizer-rewrite found no new valuable findings; tests-session-config raised TSC-1, which was accepted as MAIN-1 and submitted as the inline comment. Final convergence round 1 ended with both subagents replying NO_NEW_VALUABLE_FINDINGS for the same ledger/comment set.

User focus: no additional user-provided review focus was supplied.

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java
@Gabriel39
Gabriel39 merged commit 6c6b729 into apache:masterJul 6, 2026
33 of 34 checks passed
github-actionsBot pushed a commit that referenced this pull request Jul 6, 2026
@hubgeterYour Name
[fix](maxcompute)Fix MaxCompute IN predicate pushdown polarity. (#65083)
a1a66b6
fix maxcompute error in predicate pushdown.
@github-actionsgithub-actionsBot mentioned this pull request Jul 6, 2026
Merged
yiguolei pushed a commit that referenced this pull request Jul 7, 2026
@github-actions@hubgeter
branch-4.1: [fix](maxcompute)Fix MaxCompute IN predicate pushdown pol…
048565c
…arity. #65083 (#65238)
Cherry-picked from #65083
Co-authored-by: daidai <changyuwei@selectdb.com>
yiguolei pushed a commit that referenced this pull request Jul 13, 2026
@hubgeter
branch-4.1:[fix](maxcompute)Fix MaxCompute IN predicate pushdown pola…
d160450
…rity (#65083) (#64358)
### What problem does this PR solve?
bp #65083
Problem Summary:
fix maxcompute error in predicate pushdown.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@yiguoleiyiguolei mentioned this pull request Sep 1, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gabriel39Gabriel39Gabriel39 approved these changes
@github-actionsgithub-actions[bot]github-actions[bot] left review comments

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/4.1.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hubgeter@hello-stephen@Gabriel39@yiguolei